Method and system for auto-enhancing photographs with shadow lift adjustments

ABSTRACT

Some embodiments of the image editing and organizing application described herein provide an automatic enhancement process that includes a shadow lift adjustment. The process takes an input image and enhances the contrast of darker parts of the image. The process uses a structure histogram to determine an amount of shadow lift adjustment to apply to the image. The process tempers this adjustment based on an International Organization for Standardization (ISO) value of the image.

CLAIM OF BENEFIT TO PRIOR APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication 61/657,800 entitled “Method and System for Auto-EnhancingPhotographs,” filed Jun. 10, 2012. The contents of U.S. ProvisionalPatent Application 61/657,800 are incorporated herein by reference.

BACKGROUND

Digital images are taken with varying degrees of skill, in non-ideallighting conditions, and with cameras of variable quality. The resultsof such variations are often images that need touching up in order tolook better. Image editing and organizing applications exist to enhancethe quality of images, however these applications often contain abewildering number of options and take time and skill to make thefullest use of them. In some cases, the people who are using imageediting and organizing applications lack the skill to properly enhance adigital image. In other instances, even people with the skill to enhancea digital image manually may not have enough time to make the multipleadjustments an image requires, or may have several images to adjust.Accordingly, there is a need for an automatic enhancement process thatautomatically makes multiple adjustments to an image.

BRIEF SUMMARY

Some embodiments of the image editing and organizing applicationdescribed herein provide a multi-stage automatic enhancement process.The process takes an input image and feeds it through multiple differentenhancement operations. The multiple enhancement operations of someembodiments are carried out in a particular order. In some embodiments,the particular order starts with an exposure adjustment, then a whitebalance adjustment, then a vibrancy adjustment, then a tonal responsecurve adjustment, and finally, a shadow lift adjustment.

In some embodiments, the exposure adjustment increases the brightness ofeach pixel in the image or decreases the brightness of each pixel in aninput image. The automated white balance enhancement of some embodimentsshifts the colors of an image to make a selected object or objects inthe image adjust their color(s) toward a favored color (e.g., moving thecolor of faces toward a preset face color). The automated vibrancyenhancement of some embodiments increases or decreases how vivid animage is. The automatic tonal response curves of some embodiments makethe dark pixels darker, the light pixels lighter, and increase thecontrast of the mid-tone pixels. The automatic shadow lift enhancementof some embodiments increases the contrast on the dark parts of theimage. In some embodiments, the automatic settings for each stage arecalculated from the image as adjusted by all previous stages. Theexposure value stage is skipped in some embodiments, for a particularimage, unless the image is a RAW image with extended data. The whitebalance stage is skipped in some embodiments, for a particular image, ifthere are no faces in the image.

The tonal response curves of some embodiments are used to map a set ofinput luminance values of an image into a set of output luminance valuesof an adjusted image. The application of some embodiments generates ablack point for adjusting the darkest pixels in an input image into trueblack (minimum possible luminance) pixels in the output image.Similarly, the application of some embodiments generates a white pointfor setting the lightest pixels in an input image into true white(maximum possible luminance) pixels in the output image. The applicationthen sets the positions of three points for adjusting the mid-tonecontrast of the input image when generating the output image. The imageediting application then adjusts the values of these three points toautomatically produce a desired tonal response curve. In someembodiments, initial calculations determine original positions andvalues for the various points and then these values are tempered toreduce or change the effects they would otherwise have on the image.

The vibrancy enhancement operation of some embodiments increases thevividness of some of the pixels in the image while shielding the skintone pixels in the image from some or all of the adjustment to thepixels. The vibrancy enhancement level is automatically calculated bythe application of some embodiments. The application of some embodimentsdetermines the vibrancy enhancement level by generating a modifiedhistogram of the saturation levels of the pixels in the image. In someembodiments, the saturation level of a pixel is the difference betweenthe highest color component value of the pixel and the lowest colorcomponent value of the pixel. In some embodiments, the modifiedhistogram is generated as though the value of any pixel whose highestcolor component value is blue or green was doubled. The applicationdetermines from statistics of the histogram what vibrancy adjustmentlevel to use.

The application of some embodiments automatically generates a settingfor a shadow lift enhancement. The setting in some embodiments is basedon statistics of one or more histograms of the image. In someembodiments, one of the histograms is a conventional luminance histogramthat counts the number of pixels at each luminance level in the image.One of the histograms in some embodiments is a cumulative luminancehistogram. In some embodiments, one of the histograms is a structurehistogram that is affected by both the luminance values of theindividual pixels and the structure (arrangement of pixels) in an image.

Multiple statistical values are derived from each histogram of the imagein some embodiments. The statistical values are then combined in variousways and fed into a formula that determines a setting from thestatistical values. The application of some embodiments gets the formulafrom a mathematical regression of multiple human determined shadow liftsettings in some embodiments. That is, in order to make the applicationof some embodiments, the programmers of the application generate sets ofstatistics and combinations of statistical values from multiple images.The sets of statistics are the same type of statistics that are derivedfrom automatically adjusted images by a finished application. Theprogrammers then match a human determined setting for shadow liftingeach image to each set of statistics. The programmers then run amathematical regression of the results. The mathematical regressiongenerates the formula used by the application to automatically set theshadow lift setting level.

The application of some embodiments takes the automatically determinedvalue for the shadow lift setting and reduces it still further by anamount that depends on the ISO of the image. In some embodiments, theshadow lift operation is then performed by using a variable gammaadjustment on each pixel. The variable gamma adjustment is dependent onthe automatic setting and a Gaussian blur of the input image in someembodiments. In some embodiments, the shadow lift is either not appliedor is only lightly applied to areas of the image with skin color inthem.

The preceding Summary is intended to serve as a brief introduction tosome embodiments described herein. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description and the Drawings is needed.Moreover, the claimed subject matters are not to be limited by theillustrative details in the Summary, Detailed Description and theDrawings, but rather are to be defined by the appended claims, becausethe claimed subject matters can be embodied in other specific formswithout departing from the spirit of the subject matters.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 illustrates an auto-enhancement process of an image editing andorganizing application of some embodiments.

FIG. 2 conceptually illustrates a process for automatically enhancing animage.

FIG. 3 conceptually illustrates a software architecture diagram for aportion of an application of some embodiments.

FIG. 4 illustrates sequential images automatically generated by anapplication of some embodiments.

FIG. 5 conceptually illustrates a process of an alternate embodimentthat performs a different set of image enhancements.

FIG. 6 conceptually illustrates a process that automatically enhances animage with automatic settings based on the original image.

FIG. 7 conceptually illustrates a process for automatically adjustingthe exposure of an image.

FIG. 8 shows two versions of a RAW image with extended data before andafter an automatic exposure adjustment.

FIG. 9 illustrates a graph of a pixel brightness histogram of the RAWimage represented by the original image of FIG. 8.

FIG. 10 conceptually illustrates a process of some embodiments forperforming white balancing operations on images.

FIG. 11 illustrates controls of some embodiments set after an automaticenhancement operation.

FIG. 12 conceptually illustrates a process of some embodiments forautomatically setting locations along the x-axis for points that definetonal response curves which will be used in automatic enhancement of animage.

FIG. 13 illustrates two curves for remapping luminance values of animage.

FIG. 14 conceptually illustrates a process of some embodiments forautomatically setting the values along the y-axis for points that definethe tonal response curves which will be used in automatic enhancement ofan image.

FIG. 15 illustrates an adjustment of a lower mid-tone contrast point ofa tonal response curve of some embodiments.

FIG. 16 illustrates an adjustment of an upper mid-tone contrast point ofa tonal response curve of some embodiments.

FIG. 17 illustrates an adjustment of mid-tone contrast points closer toa baseline.

FIG. 18 conceptually illustrates a process of some embodiments forautomatically adjusting a vibrancy value.

FIG. 19 illustrates the selection of multiple pixels and how each isadded to a histogram.

FIG. 20 illustrates a modified histogram used for calculating apercentile of luminance values on the histogram.

FIG. 21 illustrates a graph of percentile location versus automaticvibrancy settings.

FIG. 22 illustrates the differences between structure histograms of someembodiments and traditional histograms for two different images.

FIG. 23 conceptually illustrates a process of some embodiments forautomatically generating a shadow lift enhancement input value.

FIG. 24A conceptually illustrates the derivation of an equation forautomatically determining a setting for shadow lifting.

FIG. 24B conceptually illustrates the use of the derived function withthe statistics of a current structure histogram.

FIG. 25 illustrates the effects of too high a shadow lift setting on animage with a high ISO.

FIG. 26 conceptually illustrates the inputs that go into generating ashadow image.

FIG. 27 conceptually illustrates a process of some embodiments forapplying or not applying skin tone protection in a shadow liftingoperation.

FIG. 28 illustrates an image to be adjusted with a low shadow liftsetting.

FIG. 29 illustrates an image to be adjusted with a high shadow liftsetting.

FIG. 30 is an example of an architecture of a mobile computing device.

FIG. 31 conceptually illustrates another example of an electronic systemwith which some embodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following detailed description of the invention, numerousdetails, examples, and embodiments of the invention are set forth anddescribed. However, it will be clear and apparent to one skilled in theart that the invention is not limited to the embodiments set forth andthat the invention may be practiced without some of the specific detailsand examples discussed.

I. Auto-Enhancement

A. Overview

The image editing and organizing applications of some of the embodimentsdisclosed herein automatically enhance images in multiple stages. Eachstage enhances a different aspect of the image. Some stages affectluminance of various pixels, some affect color, and some affect both.The combination of the stages improves the images more than theindividual enhancements would alone.

FIG. 1 illustrates an auto-enhancement process of an image editing andorganizing application of some embodiments. The figure illustrates theauto-enhancement process in 6 stages, 101-106. The first stage 101 showsan abbreviated view of the graphical user interface (GUI) of theapplication with an auto-enhance button 110 being activated. The next 4stages 102-105 show adjusted images 127, 137, 147, and 157 generated bysequential automatic enhancements performed on an original image 117.The last stage 106 shows the GUI controls and image 167 after all theauto-enhancement stages are complete. Each of these stages will befurther described below.

The first stage 101 in FIG. 1 illustrates a GUI of an image editing andorganizing application of some embodiments. The GUI includes anauto-enhance button 110, additional controls 112, and a display area 115with original image 117 displayed in it. In the illustrated embodiment,the auto-enhance button 110 activates a series of auto-enhancementstages that each alters the image sequentially. However, it will beobvious to one of ordinary skill in the art that in other embodiments,different controls are used to activate the auto-enhance stages. Forexample, in some embodiments a hot key is used. In other embodiments,the auto-enhancement is performed automatically upon the loading of animage.

The subsequent stages 102-105, do not show the GUI controls 110 and 112,but only the adjusted images. The lack of visible controls in thosestages conceptually illustrates that no user input is accepted duringthe automatic stages in the illustrated embodiment. In some embodiments,the intermediate stages of the image enhancement process are not shownto users, only the original image and the final result. In someembodiments, only when the last auto-enhancement stage is complete doesthe application again allow the user to make further changes in theimage (e.g., after stage 106).

In stage 102, the image editing and organizing application has adjustedthe original image 117. The adjustment changes an exposure setting forthe image 117 to produce image 127. Changing the exposure of the image117 multiplies one or more values of each pixel in the image 117 by aset multiplier to produce the corresponding pixels in the image 127. Inthis case, the multiplier is a value less than one, so all the pixels inthe image 127 are darker than the corresponding pixels in image 117. Theexposure auto-enhancement stage 102 is further described with respect toFIGS. 7-9.

After the exposure adjustment of stage 102, the applications of someembodiments perform a white-balancing operation in stage 103 to produceimage 137. The white-balancing operation of some embodiments (1)identifies one or more parts of the image 127, (2) determines an idealcolor for the identified part(s) and (3) adjusts the colors of the image127 in a way that changes the color of the identified part(s) toward theideal color in image 137. In stage 103, the application has determinedthat the face of the person in the back seat should be more yellow.Accordingly the entire image 127 is adjusted toward yellow to produceimage 137. The white-balance auto-enhancement stage 103 is furtherdescribed with respect to FIG. 10.

Stage 104 is a vibrancy enhancement stage. The vibrancy enhancementstage increases the color saturation of most or all of the pixels in theimage 137 to produce image 147. In some embodiments, the colorsaturation of a pixel is defined as the difference between the highestcolor component value (e.g., highest of red (r), green (g), or blue (b))and the lowest color component value (e.g., lowest of r, g, and b) ofthe pixel. The application of some embodiments raises the saturationvalue of some or all of the pixels when producing image 147 based on ahistogram of the saturation of the pixels of image 137. Increasing thesaturation values of the pixels makes the image more colorful (e.g.,makes the colors more vivid). The vibrancy auto-enhancement stage 104 isfurther described with respect to FIGS. 18-21.

In stage 105, the application remaps the luminance values of the image147 in order to make the dark pixels (e.g., the pixels in the tires) inimage 147 darker in image 157 and to make the light pixels (e.g., thepixels in the background) in image 147 lighter in image 157 and also toincrease the contrast of the mid-tone pixels in image 157 relative tothe contrast of the mid-tone pixels in image 147. This remapping isaccomplished in some embodiments by applying a remapping curve (notshown) to the values of the pixels of the image 147 to produce image157. The remapping curve enhancement stage 105 is further described withrespect to FIGS. 12-17.

Finally, the application of some embodiments finishes in stage 106 byproducing image 167 which is produced by lifting the shadows of image157. Shadow lifting increases the contrast of the darker parts of theimage to increase the visibility of details. For example, the wheels inimage 167 have spokes 169 that are highly visible in image 167. Thewheels in the other images 117, 127, 137, 147, and 157 in the otherstages 101-105 have spokes that are barely visible. The applicationincreases the contrast of the dark tires bringing the spokes 169 intoview in the dark tires. The application brings the spokes into view witha shadow lifting operation performed on image 157 of stage 105 toproduce image 167. In some embodiments, after the application performsthe multi-stage auto-enhancement, the GUI is ready to receive usercommands to adjust the image further. Accordingly, as theauto-enhancement is finished in stage 106, the user interface is shownagain in stage 106. The shadow lift enhancement stage 106 is furtherdescribed with respect to FIGS. 22-29.

B. Automatic Adjustments

The application of some embodiments selectively performs adjustments toan image. That is, the application of some embodiments either performs aparticular adjustment or skips that adjustment based on characteristicsof an image. FIG. 2 conceptually illustrates a process 200 forautomatically enhancing an image. The process 200 performs thecalculations for each stage of the auto-enhancement of the image. In theillustrated process 200, the calculations for each stage are calculatedbased on the image as adjusted by all previous stages. FIG. 2 will bedescribed in relation to FIGS. 3 and 4, which will be introduced now.

FIG. 3 conceptually illustrates a software architecture diagram 300 fora portion of an application of some embodiments. The softwarearchitecture diagram 300 shows only those portions of the applicationthat perform the auto-enhancement operation. FIG. 3 includes imagestorage 305, exposure calculator 310, exposure adjustor 312,white-balance calculator 320, white-balance adjustor 322, vibrancycalculator 330, vibrancy adjustor 332, curves calculator 340, curvesadjustor 342, shadow lift calculator 350, and shadow lift adjustor 352.The image storage 305 stores an image to be adjusted. In someembodiments, the same storage 305 stores the images before and after theauto-enhancement process. In some embodiments, the image storage 305also stores the intermediate adjusted versions of the image.

In other embodiments, different storages store the images at differenttimes. For example, in some embodiment, the original and/or final imagesare stored on a hard drive or in persistent memory (e.g., flash memory)while intermediate images are stored in Random Access Memory (RAM). Anadjusted image is passed from each adjustment module to the nextadjustment module in the sequence. As mentioned above, in thisembodiment the application calculates a setting of the automaticenhancement operation to apply to the image at each stage based on theresults of the previous stage. Accordingly, the adjusted image from eachadjustor module (of adjustor modules 312, 322, 332, and 342) is not onlypassed to the next adjustor module, but is also passed to thecalculating module that corresponds to the next adjustor module.Receiving the adjusted image allows the calculating module to deriveautomatic settings for the adjustor module based on the previouslyadjusted image rather than deriving the automatic settings from theoriginal image. The individual operations of the modules will bedescribed below in relation to the process 200 of FIG. 2.

The software architecture diagram of FIG. 3 is provided to conceptuallyillustrate some embodiments. One of ordinary skill in the art willrealize that some embodiments use different modular setups that maycombine multiple functions into one module though the figure showsmultiple modules, and/or may split up functions that the figure ascribesto a single module into multiple modules, and/or may recombine the splitup functions in various modules.

FIG. 4 illustrates sequential images automatically generated by anapplication of some embodiments. The sequential images show the actualprogression of an image as it is adjusted in multiple stages. The imagesinclude the original image 410, the final image 460, and each of theintermediate images 420-450 generated at each stage of theauto-enhancement process. Image 410 is the original image before anyenhancements have been performed. Image 420 is the image after theexposure enhancement has been performed. Image 430 is the image afterboth the exposure enhancement and the white-balance enhancements havebeen performed. Image 440 represents the image after the previouslymentioned enhancements and the vibrancy enhancement have been performed.Image 450 shows the state of the image after a tonal response curve hasbeen applied to the image 440. Finally, image 460 includes all theprevious enhancements plus a shadow lift enhancement. The followingsubsections describe each of the enhancements in more detail.Subsection 1. describes the Exposure enhancement. Subsection 2.describes the white balance enhancement. Subsection 3. then describesthe vibrancy enhancement. That is followed by a description of tonalresponse curves in Subsection 4. Subsection 5. describes shadow liftenhancement. Finally, subsection 6. describes alternate sequences ofenhancements.

1. Exposure Enhancement

As FIG. 2 shows, the process 200 begins by determining (at 205) whetherthe image is a RAW image or not. A RAW image file has data that islightly processed (almost unprocessed) from an image sensor of a digitalcamera or other digital image capturing machine. If the image is not inRAW format, then the process 200 skips the exposure enhancement stepsand proceeds to operation 225. In some embodiments the nature of theimage (RAW or not) is determined by the exposure calculator 310 of FIG.3. In other embodiments, a separate module (not shown) makes thedetermination.

If the process 200 determines (at 205) that the image is in a RAWformat, then the process 200 calculates (at 210) settings toauto-enhance the exposure. In some embodiments, this calculation isperformed by the exposure calculator 310 of FIG. 3. To begin performingthe calculation, the exposure calculator 310 of some embodimentsdetermines an average luminance value of the image. The exposurecalculator 310 then compares the average luminance to a target averageluminance. If the average luminance is higher than the target luminance,then the exposure calculator 310 determines a negative value (whichwould make the image darker) for the exposure adjustment. If the averageluminance is lower than the target luminance, then the exposurecalculator 310 determines a positive value (which would make the imagebrighter) for the exposure adjustment. If the average luminance is equalto the target luminance, then the exposure calculator 310 determines azero value (i.e., no change in exposure level) for the exposureadjustment.

In some embodiments, if the process 200 determines (at 215) that thecalculated exposure adjustment is not negative then the process skipsthe exposure adjustment operations and goes to operation 225. If theprocess 200 determines (at 215) that the calculated exposure adjustmentis negative then it performs (at 220) the exposure enhancement. In someembodiments, the exposure enhancement is performed by the exposureadjustor 312 of FIG. 3. The exposure adjuster 312 receives the originalimage (e.g., image 410 of FIG. 4) from image storage 305 and theautomatically calculated exposure value from the exposure calculator310. The exposure adjustor 312 then reduces the brightness of everypixel in the image by a factor (M) based on the automatically calculatedexposure adjustment sent to it from the exposure calculator 310. In someembodiments, the image is edited in a red, green, and blue colorcomponent format with the exposure adjustment adjusting the level ofeach color component. That is, in the applications of some embodiments,each pixel is stored as a set of red (r), green (g), and blue (b) values(a format called RGB) and the exposure adjustment multiplies the valueof r, g, and b in each pixel by a fixed multiplier (M) to generate newvalues of r, g, and, b for that pixel as shown in equation (1).(r _(new) ,g _(new) ,b _(new))=(M*r _(old) ,M*g _(old) ,M*b _(old))  (1)

In equation (1), r_(new), g_(new), and b_(new) are the values of r, g,and b for a pixel in the adjusted image, r_(old), g_(old), and b_(old)are the values of r, g, and b of the corresponding pixel in the originalimage, and M is the multiplier. This has the effect of adjusting thebrightness of the image 117 upward if M is greater than 1 (correspondingto a positive exposure adjustment) and downward if M is less than 1(corresponding to a negative exposure adjustment).

The adjustment is toward a darker image in the embodiment of FIG. 2because non-negative exposure adjustments are skipped in the process200. That is, for process 200, the adjustment (if it is performed atall) is toward a darker image, so M is less than 1. In otherembodiments, the exposure adjustment might be skipped for negativeexposure adjustments, rather than for non-negative exposure adjustments,or not skipped for either positive or negative exposure adjustments. Insuch embodiments, the value of M can be greater than 1 and still beperformed by a process similar to process 200. The described embodimentadjusts the exposure in an RGB color format. However, other embodimentsadjust the exposure in other color formats. An example of reduction inbrightness can be seen in image 420 of FIG. 4, which is darker thanoriginal image 410.

2. White Balance Enhancement

After adjusting the exposure (or after skipping the exposureadjustment), the process 200 then determines (at 225) whether there areany faces in the image 420. In some embodiments, the white balancecalculator 320 of FIG. 3 performs this search after receiving anadjusted image from exposure adjustor 312. In other embodiments, aseparate module (not shown) is provided for searching images for faces.If the process 200 determines (at 225) that there are no faces in theimage, then the process 200 skips the white balancing operation andproceeds to operation 240.

If the process 200 determines (at 225) that there are faces in theimage, then the process 200 calculates (at 230) a white balanceadjustment based on one or more faces found in the image. In someembodiments, the white balance calculator 320 begins to calculate thewhite balance adjustment by determining an average color of the faces inthe image. The white balance calculator 320 then determines a distanceand direction in a color-space between that average color and a presetcolor. The white balance calculation and adjustment is further describedin relation to FIG. 10 in section I.E., below.

The process 200 then performs (at 235) the white balancing operation. Insome embodiments, the white balance adjustor 322 (FIG. 3) moves thecolors of most or all the pixels in the image in the direction of thecalculated color difference passed to it from the white balancecalculator 320. The results of a white balancing operation can be seenin image 430 of FIG. 4. Although the determination of the color shift isbased on faces in this embodiment, the actual color shift is applied toall colored items in the image. That is, as image 430 (as compared toimage 420) shows, all the colorful parts of the image have slightlychanged colors, not just the faces.

3. Vibrancy Enhancement

Once the white balance operation has been performed (or skipped), theprocess 200 calculates (at 240) settings for increasing a vibrancy ofthe image 430. The vibrancy calculator 330 (FIG. 3) performs thecalculations in some embodiments. The calculations in some embodimentsare based on the adjusted image produced by the white balance adjustor322. In some embodiments, the vibrancy calculator 330 begins bygenerating a histogram of the saturation of pixels in the adjustedimage. The saturation of a pixel in some embodiments is defined as thedifference between the highest color component value of the pixel andthe lowest color component value of the pixel. The vibrancy calculator330 then uses statistics from the histogram to determine an amount toincrease the saturation level of the pixels. In some embodiments, thevibrancy calculator 330 also determines areas of the image that will nothave their saturation increased. Vibrancy calculations and enhancements,including variations on the histogram, are described further in sectionIII, below.

The process 200 then performs (at 245) the saturation adjustment. Thevibrancy adjustor 332 performs the adjustment of the image received fromthe white balance adjustor 322, in some embodiments, based on vibrancysettings provided by the vibrancy calculator 330. Image 440 (of FIG. 4)shows the effects of vibrancy adjustment on the image 430. Specifically,the vibrancy adjustment increases the vibrancy of the colors of some orall areas of the image.

4. Luminance Curve Enhancement

After adjusting the vibrancy, the process 200 calculates (at 250)settings to adjust the luminance values of each of the pixels in theimage with a tonal response curve (sometimes referred to as an “scurve”). The tonal response curve of some embodiments takes theluminance of each pixel of an input image and remaps it onto differentvalues in an output image by darkening the dark pixels, lightening thelight pixels and increasing the contrast of the mid-tone pixels. Theluminance values of the image are changed according to a remapping curvethat relates input luminance to output luminance. The curves calculator340 (of FIG. 3) calculates the remapping curve in some embodiments basedon the adjusted image received from the vibrancy adjustor 332. The curvecalculations are described in more detail with respect to FIGS. 12-17 insection II, below.

After the remapping curve is calculated, the process 200 then performs(at 255) the remapping curve adjustment. The curves adjustor 342performs the adjustment of the image received from the vibrancy adjustor332. The adjustments in some embodiments are based on remapping curvesettings provided by the curves calculator 340. Image 450 (of FIG. 4)shows the effects of the remapping curve adjustment on the image 440.The dark areas of image 440 are even darker in image 450. Similarly, thelight areas of image 440 are lighter in 450 and the mid-tone areas haveincreased contrast.

5. Shadow Lift Enhancement

After the tonal response curve, the process 200 of some embodimentscalculates (at 260) settings for a shadow lifting operation. In someembodiments, the shadow lifting calculations are performed by a shadowlift calculator 350 (of FIG. 3). The shadow lift calculator 350 basesthe shadow lift calculations on an adjusted image received from thecurves adjustor 342. The shadow lift calculator 350 of some embodimentsgenerates a histogram that represents the structure of the adjustedimage. In some embodiments, the shadow lift calculator 350 generatesmultiple histograms of different types. The shadow lift calculator 350then uses statistics from the histogram(s) to automatically determine asetting for the shadow lift adjustor 352.

The process 200 then performs (at 265) the shadow lift adjustment. Insome embodiments, the shadow lift adjustment is performed by the shadowlift adjustor 352. The shadow lift adjustor 352 generates a blurredversion of the image it receives and performs a variable gammaadjustment of the received image based on the blurred image and on thesetting it receives from the shadow lift calculator 350. In someembodiments, the shadow lift calculator performs some of theseoperations. The shadow lift calculations and the shadow lift adjustmentare further described with respect to FIGS. 22-29 in section IV, below.

Image 460 (in FIG. 4) shows the combined effect of all theauto-enhancements. The dark areas of the image are more defined and moredetails are visible in them than in image 450.

6. Alternate Sequences

The above described sequence of auto-enhancement, in that specificorder, is used by the applications of some embodiments. However, one ofordinary skill in the art will understand that in other embodiments, thedescribed enhancements may be performed in other orders. Furthermore,some embodiments may perform a subset of the described automaticenhancement steps. FIG. 5 conceptually illustrates a process 500 of analternate embodiment that performs a different set of imageenhancements. The process 500 does not perform operations 205-220 or240-245. Accordingly, the process 500 does not perform the exposureadjustment or vibrancy adjustment. The process 500 does performoperations 225-235 and 250-265. As in FIG. 2, in FIG. 5 operations225-235 automatically calculate and perform a white balance adjustmentif there are faces in the image. Operations 250 and 255 automaticallycalculate and perform tonal response curve adjustment. Operations 260and 265 automatically calculate and perform a shadow lift adjustment.

C. Pre-Calculated Auto-Adjustments

As described above, the image editing application of some embodimentsautomatically enhances images in multiple sequential stages, each ofwhich enhances the image in a different way. In the above describedembodiments, each sequential stage is performed on an image that hasalready been adjusted by all of the previous stages. Furthermore, notonly were the adjustments made sequentially, but the calculations thatdetermined the automatic settings were also made sequentially. Eachautomatic setting was calculated based on the image as it was after ithad been adjusted by all of the previous stages. In contrast, in someembodiments, the application automatically determines what settings touse for each of the stages based on the original image.

FIG. 6 conceptually illustrates a process 600 that automaticallyenhances an image with automatic settings based on the original image.The process 600 performs many of the same operations as the process 200,where the operations are sufficiently similar, the operations will bereferred to by reference to the operations of process 200 from FIG. 2.The process 600 calculates (at 610) all the settings to automaticallyapply at every stage of auto-enhancement. The calculations automaticallydetermine, based on the original image, settings for an exposureadjustment, a color balance adjustment, a vibrancy adjustment, aluminance curve adjustment, and a shadow lift adjustment. Aftercalculating the settings, the process then performs (at 220) an exposureadjustment. The process then performs (at 235) a white balanceadjustment on the image. After the white balance adjustment, the process600 performs (at 245) a vibrancy adjustment and then performs (at 255) aremapping curve adjustment. Finally, the process 600 performs (265) ashadow lift adjustment.

The process 600 performs the same type of enhancements as the process200. However, one of ordinary skill in the art will understand thatother embodiments are possible within the scope of the invention. Forexample, some embodiments provide an image editing and organizingapplication with a pre-calculating process that does not include theautomatic adjustment of exposure. Additionally, the applications of someembodiments perform the adjustments in a different order.

D. Automatic Exposure Enhancement

As described above, the image editing and organizing applications ofsome embodiments perform auto-enhancements that include automaticexposure adjustments. The above described applications onlyautomatically adjust the exposure under certain circumstances, such aswhen the calculated exposure value is negative and the image is a RAWimage. The applications of some other embodiments place even morerestrictions on use of the automatic exposure adjustment. For example,the applications of some embodiments determine whether or not to useautomatic exposure adjustment based on whether the RAW image hasextended data. Extended data is possible because RAW images are storedin a wider gamut than most processed images. Therefore, areas of animage that would be displayed as uniformly 100% pure white in a narrowergamut color space can have details in the RAW image format.

FIG. 7 conceptually illustrates a process 700 for automaticallyadjusting the exposure of an image. FIG. 7 will be described withrespect to FIGS. 8 and 9 which will be described briefly now and in moredetail in context. FIG. 8 shows two versions of a RAW image withextended data before and after an automatic exposure adjustment. Thefigure includes original image 810 and adjusted image 820. Originalimage 810 is a narrow gamut representation of a RAW image. Adjustedimage 820 is another narrow gamut image, but represents the RAW imageafter its exposure has been reduced.

FIG. 9 illustrates a graph of a pixel brightness histogram 900 of theRAW image represented by original image 810 of FIG. 8. FIG. 9 includesthe histogram 900, a white point 910, and an extended percentage ofpixels 920. The histogram 900 is a graph with the brightness (e.g.,luminance) of pixels on the x-axis and the number of pixels with thatparticular brightness on the y-axis. The white point 910 is the pointalong the x axis at which the standard range ends. The extendedpercentage of pixels 920 includes pixels that are brighter (i.e., havehigher numerical value) than the white point. In the display of theoriginal image 810, all the pixels at or beyond the white point areshown as 100% luminance (i.e., pure white pixels).

The process 700 begins (at 705) to automatically enhance the image. Theprocess 700 determines (at 710) whether the image is a RAW image. If theimage is not a RAW image then the process 700 leaves (at 715) theexposure setting unchanged and ends. If the image is a RAW image, thenthe process 700 calculates (at 720) a histogram of the RAW image. Asmentioned above, original image 810 of FIG. 8 is a narrow gamutrepresentation of a RAW image. Histogram 900 of FIG. 9 is a histogram ofthat RAW image. Pixels in the extended percentage of pixels 920 area ofthe histogram are represented as pure white pixels in the original image810. Any detail in the data representing the pixels in the extendedrange is not shown in original image 810 because all pixels in thatrange are shown in original image 810 as uniformly white.

The applications of some embodiments, when performing a multistageauto-enhancement, automatically change the exposure level only whenthere are pixels in the extended range. Process 700 determines (at 725)whether there are pixels in the extended data range. If there are nopixels in the extended data range, then the process 700 goes tooperation 715, leaving the exposure setting unchanged, and then ends.

If there are pixels in the extended data range, then the process 700calculates (at 730) an exposure setting. To calculate the exposuresetting, the application of some embodiments determines an averageluminance of the pixels in the image. The application then compares theluminance to a target luminance (e.g., 50% of the maximum possibleluminance). If the average luminance is lower than the target luminance,then the application determines a change in the exposure value that willraise the average luminance of the image toward the target value. Incontrast, if the average luminance is higher than the target luminance,then the application determines a change in the exposure value that willlower the average luminance toward the target luminance. Someembodiments use methods for calculating exposure values as described inU.S. patent application entitled “Tempered Auto-Adjusting, Image-EditingOperation” filed Jun. 10, 2012 with Ser. No. 61/657,794, and inconcurrently filed U.S. patent application Ser. No. 13/629,504, nowpublished as U.S. 2013/0332866, entitled “Tempered Auto-Adjusting,Image-Editing Operation ”. Both of these Applications are incorporatedherein by reference.

Once an exposure value is calculated, the process determines (at 735)whether the exposure value is negative and determines (at 740) whetherthe exposure value is beyond a particular threshold value (e.g., whetherthe calculated exposure value is less than −0.005). If the exposurevalue is positive then the exposure value setting is left (at 715)unchanged and the process ends. Similarly, if the exposure value isnegative, but not negative enough to be beyond the threshold value thenthe exposure value is left (at 715) unchanged. If the exposure value isnegative and less than the threshold value then the process 700 adjusts(at 745) the exposure setting (downward). The result of such a reductionin exposure value setting can be seen in adjusted image 820 of FIG. 8.In adjusted image 820, areas that contained blown out white pixels nowshow more details and the pure white areas have been reduced to smallerhighlights. That is, the detail that was there in the extended rangedata, but not visible in original image 810 has been made visible inadjusted image 820.

One of ordinary skill in the art will understand that the conditionaldeterminations as shown in FIG. 7 can be performed in other ways and inother orders in other embodiments. For example, in some embodiments,multiple conditional statements are made together. For instance, in someembodiments the application combines operations 735 and 740 and simplydetermines whether the calculated exposure value is less than aparticular negative value rather than separately checking whether thecalculated exposure value is negative and then whether it is less thanthat value.

Additionally, some embodiments may make some determinations implicitly,rather than using an explicit conditional statement. For example, thedetermination shown as operation 725 may be made implicitly. That is,some embodiments automatically generate an exposure value of zero (whichdoes not change the input image) as a consequence of the lack of data inthe extended data range. For example, some embodiments generate anexposure value (e.g., as in operation 730) and use an equation such asequation (2) to modify the exposure value calculated in operation 730.final_exposure=min(1.0,3.0*extended_Percent)*original_exposure  (2)

In equation (2), “original_exposure” is an initial exposure valuesetting as calculated by the application (e.g., using the methods ofU.S. Provisional Patent Application 61/657,794). “Extended_Percent” isthe percentage of pixels that are in the extended data range (e.g., thepixels represented by the extended percentage of pixels 920 in FIG. 9).“Final exposure” is the exposure setting used in operation 745 to adjustthe image. The use of equation (2) ensures that if there is no extendedrange data, that the exposure adjustment will be zero because if“extended_Percent” is zero then “final_exposure” is zero. In someembodiments, rather than the percentage of pixels in the extended datarange, the “extended_Percent” represents the percentage by which thehighest luminance pixel exceeds the non-extended range scale. Equation(2) sets the final exposure value setting to be either smaller or equalin magnitude to the initially determined exposure value setting.

E. Automatic White Balance Enhancement

The image editing and organizing application of some embodimentsperforms automatic white balancing operations. The application of someembodiments performs a white balancing operation only if there are facesin the image. In some embodiments, the application converts the imageinto a different color space from the original color space of the image,performs the color adjustments in the different color space, thenconverts back to the original color space.

FIG. 10 conceptually illustrates a process 1000 of some embodiments forperforming white balancing operations on images. The process 1000automatically performs a white balancing operation based on faces in animage (if any). The process 1000 begins by searching (at 1005) the imagefor faces. In some embodiments, this searching is performed by standardface searching techniques. The process then determines (at 1010) whetherany faces have been found in the search. If no faces have been found inthe image the process 1000 ends (and in some embodiments, the next autoenhancement operations will be performed). On the other hand, if a faceis found in the image, then the process 1000 determines (at 1015)whether the image is in a wide gamut format or not. If the image is notin a wide gamut format the process 1000 converts (at 1020) the imageinto a wide gamut format. In some embodiments, the wide gamut format isa wide gamut RGB format, while in other embodiments, it is in some otherformats. If the image is determined (at 1015) to be in a wide gamutformat the process skips operation 1020 to go directly to operation1025.

The process 1000 then performs (at 1025) a gamma adjustment on the imageby taking each color component value of each pixel in the image andraising it to the power of 1/n, where n is a number. In someembodiments, “n” is equal to 4. In other embodiments, other numbers areused. Some embodiments use equation (3) to perform the gamma adjustment.(r _(new) ,g _(new) ,b _(new))=(r _(old)^¼,g _(old)^¼,b _(old)^¼)  (3)

In equation (3), r_(new), g_(new), and b_(new) are the values of r, g,and b for a pixel in the gamma adjusted image. In equation (3), r_(old),g_(old), and b_(old) are the values of r, g, and b of the correspondingpixel in the input image.

The process then converts (also at 1025) the gamma adjusted image to anopponent color space (sometimes called a YCC space) that includes aluminance component and two color components (e.g., a YIQ color space).After converting to the YCC color space, the process then determines (at1030) in the YCC color space the difference between the average facecolor of the image (e.g., the average color of the faces found inoperation 1005) and a preset face color. The process then identifies (at1035) a vector in color space from the average face color to the presetface color. The vector in color space has a direction in the color spaceand a magnitude in the color space that spans the difference between theaverage face color and the preset face color.

Once the color direction and magnitude are set, the process selects (at1040) a pixel in the input image. The process determines (at 1045) thechroma level of the pixel. The process adjusts (at 1050) the color ofthe pixel in the previously determined direction and by an amount thatis determined by the chroma level of the pixel (e.g., larger chromavalues of the input pixel result in larger color shifts when generatingthe output pixel) and the magnitude determined in operation 1035. Insome embodiments, pixels with zero chroma levels in the input image(e.g., gray, white, and black pixels) do not have their colors changed.In some embodiments, the magnitude of the adjustment for some pixels iscapped at the magnitude determined in operation 1035, regardless of thechroma level of the pixel before adjustment.

Once the selected input pixel has been color shifted (or has beenthrough the color shifting process with a zero color shift, such as agray input pixel), the process determines (at 1055) whether the selectedpixel was the last pixel in the image (i.e., whether all pixels havebeen through the color shifting process). If the selected pixel was notthe last pixel in the input image, then the process 1000 loops back tooperation 1040 to select the next pixel. If the process 1000 determines(At 1055) that the selected pixel was the last pixel, then the processconverts (at 1060) the image back into the wide gamut format andperforms an inverse gamma operation on the image, such as equation (4).(r _(new) ,g _(new) ,b _(new))=(r _(old)^4,g _(old)^4,b _(old)^4)  (4)

In equation (4), r_(new), g_(new), and b_(new) are the values of r, g,and b for a pixel in the inverse gamma adjusted image, r_(old), g_(old),and b_(old) are the values of r, g, and b of the corresponding pixel inthe image that has just been adjusted (in the YCC format). One ofordinary skill in the art will understand that the value of 4 for theinverse gamma power is only an example, and is used here as the inverseof the original gamma value in the example equation (3). Other gamma andinverse gamma values are possible in some embodiments. Furthermore inalternate embodiments, the inverse gamma value may not be the exactinverse of the original gamma value (e.g., to have the same effect as agamma adjustment to the final image). In some embodiments, the process1000 then converts (at 1065) the image to some other color space (e.g.,back to the original color space of the image). For example, if theimage was originally a non-wide gamut RGB image, the process 1000 ofsome embodiments converts it back to that RGB format before ending. Moredetails on the white balancing operations using faces can be found inU.S. patent application Ser. No. 13/152,206, now issued as U.S. Pat. No.8,565,523, entitled “Image Content-Based Color Balancing”, U.S.Provisional Patent Application 61/657,795 entitled “Color Balance Toolsfor Editing Images” filed Jun. 10, 2012, concurrently filed U.S. patentapplication Ser. No. 13/629,529, now published as U.S. 2013/0329994,entitled “Color Balance Tools for Editing Images”, concurrently filedU.S. patent application Ser. No. 13/629,480, now issued as U.S. Pat. No.8,885,936, entitled “Automated Color Balance Tools for Editing Images”,and in concurrently filed U.S. patent application Ser. No. 13/629,496,now published as U.S. 2013/0328906, entitled “Gray Color Balance Toolsfor Editing Images”. All of the above-mentioned Applications areincorporated herein by reference.

While the application of the above described embodiments adjusts thewhite balance only if there are faces in the image, in otherembodiments, the white balance is automatically adjusted whether or notthere are faces in the image. In some such embodiments, the applicationuses a gray edge assumption to perform a color balancing operation,either as an alternative to or in addition to performing white balancingoperations.

In the gray edge assumption, the application uses the assumption thatthe edges of objects are more likely to reflect the color of the lightthan the general surface of the objects. The application of someembodiments therefore determines an average color of the edges of theobjects in the image and shifts the colors of all the pixels in theimage in such a way as to move the average color of the edges towardgray. In some embodiments, the application tempers the color shift inaccordance with the luminance of the pixel. For example, in someembodiments, the color shifts for darker pixels are less than the colorshifts for lighter pixels. In other embodiments, the color shifts formedium luminance pixels are greater than the color shifts for eithervery dark or very bright pixels.

The tempering of the color shift based on luminance of the pixels isdifferent from the tempering used in the skin tone based white balancingoperation which uses the chroma values of the pixels to temper the coloradjustment. Unlike the skin tone based white balance operation, the grayedge assumption of some embodiments does not preserve existing grays.That is, a skin tone based white balance leaves existing grays as grays,while a gray edge assumption based white balance operation shifts thecolors of gray pixels as well, in some embodiments. However, in otherembodiments, the gray edge based white balance operation may use chromavalues of the pixels to temper the color adjustments and thus preserveexisting grays.

The application of some embodiments performs the gray edge based whitebalance operation during an auto-enhancement operation only when thereare no faces in the image. Applications of other embodiments performgray edge based white balance operations even when there are faces inthe image. Applications of still other embodiments may perform multiplewhite balance operations in one auto-enhancement operation, such asperforming a gray edge based white balance operation followed by a skintone based white balanced operation. Furthermore, some embodiments mayuse a gray world assumption rather than or in addition to using a grayedge assumption. A gray world assumption based white balance operationdetermines an average color of the entire image instead of an averagecolor of the edges. The operation then adjusts the colors of the pixelsto move the average color of the entire image toward gray. In someembodiments, the color shift in a gray world based white balanceoperation is also tempered based on the luminance values of the pixels.

F. Automatic Control Settings

In some embodiments, when setting the automatic enhancement levels ofthe various automatic enhancement stages, the application also setsenhancement controls to levels matching the levels of the automaticenhancements. FIG. 11 illustrates controls of some embodiments set afteran automatic enhancement operation. For space considerations, the figurecontains controls for only four of the listed stages. In the applicationof some embodiments, the controls for the tonal response curves areimages of the tonal response curves, such as will be shown in FIG. 17,in section II.B., below.

In some embodiments, the various points of the tonal response curve canbe adjusted manually. FIG. 11 includes exposure controls 1110, whitebalance controls 1120, vibrancy controls 1130, and shadow lift controls1140. The exposure controls 1110 allow a user to set the exposure valuemanually, including changing the automatically set exposure value. Thewhite balance controls 1120 allow a user to set the white balancemanually, including changing the automatically selected face areas andchanging the warmth of the white balance adjustment. The vibrancycontrols 1130 allow a user to set the vibrancy setting manually,including changing the automatically set vibrancy setting. The shadowlift controls 1140 allow a user to set the shadow lift manually,including changing the automatically set shadow lift setting. In someembodiments, each of the controls 1110-1140 is set automatically to asetting or settings that reflect the automatic adjustments that havebeen made in the automatic enhancement stages corresponding to thatcontrol. However, the application of some embodiments leaves one or morecontrols in a neutral, default position (e.g., a central position),despite any automatic changes to the image based on the enhancement(s)set by those controls. Such controls are left in the default position togive the user a full range of options for further modifications of theimage. For example, in the application of some embodiments, the whitebalance warmth control (in control set 1120) is left at the center ofits range in order to allow the user to increase or decrease the warmthwithout being limited by the preceding automatic adjustment.

II. Tonal Response Curves

Tonal response curves themselves are known in the art, but the known artdoes not produce such curves automatically in the manner of theapplication of some embodiments. The application of some embodimentsuses tonal response curves to selectively adjust the luminance of thevarious pixels in an input image. The tonal response curves remap theluminance of pixels in an input image to new luminance values in thepixels of an output image. FIGS. 12-17 will be used to describe theautomatic production of the tonal response curves of some embodiments.For convenience, the x-axis coordinates of points may be referred toherein as “locations” while the y-axis coordinates of points may bereferred to as “values”. FIG. 12 conceptually illustrates a process 1200of some embodiments for automatically setting locations along the x-axisfor points that define tonal response curves which will be used inautomatic enhancement of an image. FIG. 14 conceptually illustrates aprocess 1400 of some embodiments for automatically setting the valuesalong the y-axis for points that define the tonal response curves whichwill be used in automatic enhancement of an image. FIG. 12 will bedescribed in relation to FIG. 13, which will be briefly described first.

A. Setting the x-Axis Locations of Tonal Response Curve Points

FIG. 13 illustrates two curves for remapping luminance values of animage. The figure shows a contrast between a curve with untemperedsettings and a curve with tempered settings. FIG. 13 includes curves1310 and 1320. Curve 1310 includes black point 1312, white point 1314,lower mid-tone contrast point 1316, upper mid-tone contrast point 1318,and median point 1319. Curve 1320 includes black point 1322, white point1324, lower mid-tone contrast point 1326, upper mid-tone contrast point1328, and median point 1329.

Curves 1310 and 1320 conceptually illustrate remapping functions to beapplied to an image (e.g., to luminance values of the image) by theimage editing and organizing application of some embodiments. The curvesmap the luminance values of pixels of an input image (represented bylocations along the x-axes of the graphs containing the curves) toluminance values of corresponding pixels of an output image (representedby values along the y-axes of the graphs containing the curves). Forexample, if a pixel in an image has a luminance of 0.2 (or below) thenremapping that image using curve 1310 would remap that pixel to aluminance of 0 (i.e., the corresponding value along the y-axis on thecurve 1310). Similarly, remapping a pixel with a luminance of 0.2 usingremapping curve 1320 would remap that pixel to a pixel with a luminancevalue of approximately 0.1.

When tonal response curve 1310 is applied to an input image, any pixelsin that input image with a luminance level at or less than (to the leftof) the location of black point 1312 are remapped (in the output image)to luminance zero (black). The black point 1312 is automatically given avalue of zero along the y-axis and a location along the x-axisdetermined by the histogram of the image. Similarly, any pixels in theinput image with a luminance at or more than (to the right of) thelocation of white point 1314 are remapped (in the output image) toluminance one (white). The white point 1314 is automatically given avalue of one along the y-axis and a location along the x-axis determinedby the histogram of the image.

The lower mid-tone contrast point 1316, the median point 1319, and theupper mid-tone contrast point 1318 together affect the shape of thecurve 1310 between the black point 1312 and the white point 1314. Theshape of the curve in turn determines how the intermediate luminancevalues (values between the locations of the black point 1312 and thewhite point 1314) of an input image will map to luminance values in anoutput image. For curve 1310, the mid-tone contrast points 1316 and 1318and the median point 1319 are aligned with the black point 1312 andwhite point 1314, making curve 1310 a straight line. The straight lineof tonal response curve 1310 means that the output image luminance of apixel of an intermediate luminance value is a linear function of theluminance of the corresponding input pixel. The difference between theluminance values of any two intermediate luminance pixels in the outputimage will be a fixed multiple of the difference between the luminancevalues of the corresponding two pixels in the input image. In someembodiments, the fixed multiple is the slope of the curve (e.g., curve1310 or 1320).

Like the black point 1312, the black point 1322 of tonal response curve1320 defines a cutoff point. Pixels in an input image with luminancevalues to the left of the black point 1322 are remapped in the outputimage to luminance zero (i.e., black pixels) when tonal response curve1320 is applied to them. The black point 1322 has a location along thex-axis that is adjusted from the location of black point 1312, whilemaintaining the value “zero” along the y-axis. The white point 1324defines a second cutoff point. Pixels in an input image with luminancevalues to the right of the white point are remapped in the output imageto luminance one (i.e., white pixels) when tonal response curve 1320 isapplied to them. The white point 1324 has a location along the x-axisthat is adjusted from the location of the white point 1314, whilemaintaining the value “one” along the y-axis. Similar to what isdescribed above with respect to the points of curve 1310, the lowermid-tone contrast point 1326, the median point 1329, and the uppermid-tone contrast point 1328 together affect the shape of the curve 1320between the black point 1322 and the white point 1324.

1. Setting the Black Point and White Point Locations

The locations along the x-axis of the black point 1322 and white point1324 of curve 1320 are determined using process 1200 of FIG. 12. Theprocess 1200 begins by generating (at 1205) a histogram of luminancevalues for an input image. The histogram can be represented as bins ofpossible luminance locations along the x-axis and counts of the numberof pixels in each bin along the y-axis.

The histogram is analyzed (at 1210) for statistics. In some embodiments,the statistics include the location (along the x-axis) of fiveparticular percentiles of pixels. In some embodiments, the process 1200determines the 0.1, 25, 50, 75, and 99.9 percentile luminance values ofthe pixels in the input image. For example, the 0.1 percentile (e.g.,the darkest 0.1% of pixels) in an image having 1 million pixels wouldencompass the 1,000 darkest pixels of the image, the 25^(th) percentileencompasses the 250,000 darkest pixels, the 50^(th) percentileencompasses the 500,000 darkest pixels, the 75^(th) percentile includesthe 750,000 darkest pixels, and the 99.9 percentile includes the 999,000darkest pixels (or alternatively, the 99.9 percentile level has 1000pixels brighter than that level).

The percentiles are not the same as the percentage of the possibleluminance values. It is possible for the 1000 darkest pixels in an imageto all be at the lowest possible value for luminance (e.g., zero), to bespread out from the lowest possible level to some other level (e.g.,spread over 20% of the available luminance range), or even to beentirely found at levels above the minimum possible level. In someembodiments, when analyzing (at 1210) the histogram, the process 1200sets the location of the median point (e.g., median point 1319, shown inFIG. 13) to the 50^(th) percentile point of the histogram. In someembodiments, the location of the median point is then left unchanged(e.g., as is median point 1329 in FIG. 13). Although the above describedembodiments use the 0.1, 25, 50, 75, and 99.9 percentile locations inthe histogram as their particular percentiles for setting points on thetonal response curve, in other embodiments, other percentiles are used.

After the histogram of an input image is analyzed to determine the fiveparticular percentile levels and the median point, a black point iscalculated (at 1215) at the location of the lowest of the particularpercentiles (e.g., the 0.1 percentile). For example, if the 0.1percentile of the histogram is at location 0.2 along the x-axis, thenthe black point is set to a location of 0.2. This example is shown inFIG. 13 as black point 1312 of remapping curve 1310. If the input imagewere remapped with curve 1310, then the 0.1 percentile of the pixels(all those at or below 0.2 luminance in the input image) would beremapped in the output image to luminance zero (darkest possible black).

However, for some images, there may be a reason why there are so fewvery dark pixels in the image (before remapping). For example, a foggyscene may result in an image with few very dark pixels. Accordingly, theapplications of some embodiments temper the black point. Tempering theblack point reduces the number of truly black pixels in the outputimage. This is accomplished by conceptually moving the black point to alocation on the x-axis to the left of where the original histogrampercentile puts it. Therefore, process 1200 adjusts (at 1220) the blackpoint leftward. The application of some embodiments multiplies thelocation of the black point by a factor (of less than one) that isdetermined by a statistic from the histogram (e.g., from the 0.1percentile location itself). Some embodiments use equation (5) to adjustthe black point leftward.black_point=0.75*(1.0−0.65*histogram_black^0.5)*histogram_black  (5)

In equation (5) “histogram_black” is the location of the lowest of theparticular determined percentile levels, while “black_point” is thelocation on the x-axis of the adjusted black point. Black point 1322 ofFIG. 13 is the adjusted black point corresponding to a “histogram_black”value of 0.2. If the image were remapped with curve 1320, then not allof the 0.1 percentile of the pixels would be mapped to the luminancezero (darkest possible black). As a result of moving the black point,only those pixels with an input luminance less than about 0.106 (ascalculated by equation (5)) will be mapped to luminance zero (darkestpossible black).

Like the calculation of an initial black point (at 1215), the process1200 of some embodiments calculates (at 1225) an initial white pointbased on the histogram (e.g., the 99.9 percentile location). Forexample, if 99.9% of the pixels in an image have a luminance less than0.8, then the initially calculated white point will be 0.8. This exampleis shown in FIG. 13 as white point 1314 of remapping curve 1310. If theimage were remapped with curve 1310, then the top 0.1% of the pixels(all those at or above 0.8 luminance in the input image) would be mappedto luminance one (brightest possible white).

However, there may be a reason why there are so few very bright pixelsin the image (before remapping). For example, an indoor scene at nightmay result in an image with few very bright pixels. Accordingly, theapplications of some embodiments temper the white point. Tempering thewhite point reduces the number of pure white pixels in the output image.This is accomplished by conceptually moving the white point to theright. Therefore, process 1200 adjusts (at 1230) the white pointrightward. The application of some embodiments multiplies the distanceof the white point from the maximum end of the luminance scale by afactor (of less than one) that is determined by a statistic from thehistogram (e.g., from the 99.9 percentile location itself). Someembodiments use equations (6A)-(6B) to adjust the white point upwards.white_distance=0.5*min(1,max(0.6,1.0−0.8*(1−histo_white)))*(1−histo_white)  (6A)white_point=1−white_distance  (6B)

In equations (6A) and (6B) “histo_white” is the location of the highestof the particular percentile levels as calculated from the analysis ofthe histogram, “white_distance” is the distance of the adjusted whitepoint from the high end of the histogram (e.g., distance from luminancevalue of 1), “white_point” is the location of the adjusted white point.White point 1324 of FIG. 13 is the adjusted white point corresponding toa “histo_white” value of 0.8. If the image were remapped with curve1320, then not all of the top 0.1% brightest pixels (e.g., those withluminance above 0.8) would be mapped to the luminance one (brightestpossible white). As a result of moving the white point to the right,only those pixels with an input luminance greater than 0.916 (ascalculated by equations (6A) and (6B)) will be mapped to luminance one(brightest possible white).

2. Alternate Black Point Setting

As described above, the initial black point is determined using aparticular percentile of the luminance values in a luminance histogramof the input image. However, the application of some embodiments iscapable of using multiple different percentiles, depending on thecharacteristics of the image. In some embodiments, the applicationdetermines the location of the 0.01 percentile and the 0.05 percentileas well as the 0.1 percentile. If the 0.01 percentile is above the 1%luminance value (i.e., some of the lowest 0.01% of pixels in terms ofluminance values are above 1% of the full scale on the histogram) thenthe 0.01 percentile location is averaged in (as a weighted average) withthe 0.1 percentile location. The weighted average is determined usingequations (7A) and (7B)wtLow=min(1.0,(loc_hundredth−0.01)/0.03);  (7A)histogram_black_new=wtLow*loc_hundredth+(1.0−wtLow)*histogram_black;  (7B)

In equations (7A) and (7B) “wtLow” is the weighting factor thatdetermines how much of the 0.1 percentile location to use and how muchof the 0.01 percentile location to use in the weighted average.“Histogram_black” is the previously identified original location of theblack point (e.g., the 0.1 percentile location). “histogram_black_new”is the replacement for the histogram derived black point. “loc_hundredthis the location (along the x-axis) in the histogram of the 0.01percentile point. From the equations (7A) and (7B) one of ordinary skillin the art will understand that if the 0.01 percentile location isgreater than 0.04 (i.e., 4%) of the full scale of the histogram, thenweighting of the 0.01 percentile location will be 1. Therefore, if the0.01 percentile location is greater than 4% of the full scale of thehistogram only the 0.01 percentile location will be used to determinethe histogram black location. In some embodiments, the new histogramblack is then put through the same tempering equation (5) as describedabove for the histogram black derived from the 0.1 percentile location.

If the 0.01 percentile location is below 1% of the full histogram scale,but the 0.05 percentile location is above 1% of the full histogramscale, then a new histogram black location will be generated that is theweighted average of the 0.05 percentile location and the 0.1 percentilelocation. In some embodiments, the application generates the weightedaverage using equations (8A) and (8B)wtLow=min(1.0,(loc_twentieth−0.01)/0.03);  (8A)histogram_black_new=wtLow*loc_twentieth+(1.0−wtLow)*histogram_black;  (8B)

In equations (8A) and (8B) “wtLow” is the weighting factor thatdetermines how much of the 0.1 percentile location to use and how muchof the 0.05 percentile location to use in the weighted average.“Histogram_black” is the previously identified original location of theblack point (e.g., the 0.1 percentile location). “histogram_black_new”is the replacement for the histogram derived black point. “loc_twentiethis the location (along the x-axis) in the histogram of the 0.05percentile point. From the equations (8A) and (8B) one of ordinary skillin the art will understand that if the 0.05 percentile location isgreater than 0.04 (i.e., 4%) of the full scale of the histogram, thenweighting of the 0.05 percentile location will be 1. Therefore, if the0.05 percentile location is greater than 4% of the full scale of thehistogram only the 0.05 percentile location will be used to determinethe histogram black location. In some embodiments, the new histogramblack is then put through the same tempering equation (5) as describedabove for the histogram black derived from the 0.1 percentile location.

3. Setting the Mid-Tone Contrast Point Locations

The applications of some embodiments set the initial locations along thex-axis of the lower and upper mid-tone contrast points 1326 and 1328 (ofFIG. 13). The process 1200 calculates (at 1235) an initial locationalong the x-axis for the lower mid-tone contrast point. The initiallocation is somewhere between the location along the x-axis of themedian point of the histogram (e.g., median point 1319 of FIG. 13) andthe black point (e.g., black point 1312 of FIG. 13). In someembodiments, the initial location of the lower mid-tone contrast pointis the 25^(th) percentile of the histogram. In FIG. 13 the initiallocation of the lower mid-tone contrast point is at 0.25. For thehistogram values used to generate the curves 1310 and 1320, the value ofthe lower mid-tone contrast point happens to be the 0.25 luminancelocation. However, one of ordinary skill in the art will realize thatthe 25^(th) percentile is not necessarily going to be at the 0.25luminance location for all images.

After calculating (at 1235) the initial location along the x-axis of thelower mid-tone contrast point (e.g., lower mid-tone contrast point 1316of FIG. 13), the process adjusts (at 1240) the location along the x-axisof the lower mid-tone contrast point. In some embodiments, the process1200 adjusts the lower mid-tone contrast point to a location that is aweighted average of the locations of: (1) the adjusted black point(e.g., black point 1322 of FIG. 13), (2) the location (along the x-axis)of the median point (e.g., median point 1329 of FIG. 13), and (3) theinitial location of the lower mid-tone contrast point (e.g., mid-tonecontrast point 1316 of FIG. 13). In some embodiments, the greater thegap between the black point and the median point, the more heavily thosepoints are weighted in determining the adjusted location for the lowermid-tone contrast point. Some embodiments determine the adjustedlocation for the lower mid-tone contrast point (e.g., the lower mid-tonecontrast point 1326 of FIG. 13) according to equations (9A) and (9B).gap=median_location−black_point;  (9A)LMCP=(ILMCP+(1+gap)*black_point+(1+gap)*median_location)/(3+2*gap)  (9B)

In equations (9A) and (9B) “median_location” is the location along thex-axis of the median (e.g., median point 1329 of FIG. 13), “black_point”is the location along the x-axis of the adjusted black point (e.g.,black point 1322 of FIG. 13), “gap” is the distance along the x-axisfrom the adjusted black point to the median point, “LMCP” is thelocation along the x-axis of the adjusted lower mid-tone contrast point(e.g., lower mid-tone contrast point 1326 of FIG. 13), and “ILMCP” isthe initial location of the lower mid-tone contrast point (e.g., lowermid-tone contrast point 1316 of FIG. 13). In FIG. 13, as a result of thecalculations, the new adjusted mid-tone contrast point has moved to theright.

In some embodiments, the calculation of the upper mid-tone contrastpoint is similar to the calculation of the lower mid-tone contrastpoint. The process 1200 calculates (at 1245) an initial location alongthe x-axis for the upper mid-tone contrast point. The initial locationis somewhere between the location along the x-axis of the median pointof the histogram (e.g., median point 1319) and the white point (e.g.,white point 1314). In some embodiments, the initial location of theupper mid-tone contrast point is the 75^(th) percentile of thehistogram. In FIG. 13 the initial location of the upper mid-tonecontrast point is at a location 0.6 along the x-axis.

After calculating (at 1245) the initial location along the x-axis of theupper mid-tone contrast point (e.g., upper mid-tone contrast point1318), the process adjusts (at 1250) the location along the x-axis ofthe upper mid-tone contrast point. In some embodiments, the process 1200adjusts the upper mid-tone contrast point to a location that is aweighted average of the locations of: (1) the adjusted white point(e.g., white point 1324 of FIG. 13), (2) the location (along the x-axis)of the median point (e.g., median point 1329 of FIG. 13), and (3) theinitial location of the upper mid-tone contrast point (e.g., uppermid-tone contrast point 1318 of FIG. 13). In some embodiments, thegreater the gap between the white point and the median point, the moreheavily those points are weighted in determining the adjusted locationfor the upper mid-tone contrast point. Some embodiments determine theadjusted location for the upper mid-tone contrast point (e.g., the uppermid-tone contrast point 1328) according to equations (10A) and (10B).gap=white_point−median_location;  (10A)UMCP=(IUMCP+(1+gap)*white_point+(1+gap)*median_location)/(3+2*gap)  (10B)

In equations (10A) and (10B) “median_location” is the location along thex-axis of the median (e.g., median point 1329), “white_point” is thelocation along the x-axis of the adjusted white point (e.g., white point1324), “gap” is the distance along the x-axis from the adjusted whitepoint to the median point, “UMCP” is the location along the x-axis ofthe adjusted upper mid-tone contrast point (e.g., upper mid-tonecontrast point 1328), and “IUMCP” is the initial location of the uppermid-tone contrast point (e.g., upper mid-tone contrast point 1318).Here, the adjustment has moved the upper mid-tone contrast point 1328 tothe right of where upper mid-tone contrast point 1318 is. For reasons ofclarity and simplicity, the values of the median points 1319 and 1329and the upper and lower mid-tone contrast points 1316, 1318, 1326, and1328 in FIG. 13 are shown as being on the line connecting theirrespective black points 1312 and 1322 to their respective white points1314 and 1324. However, in some embodiments, some or all of the valuesare set to different levels on the y-axis depending on characteristicsof the image.

B. Setting the y-Axis Values of Tonal Response Curve Points

FIG. 14 illustrates a process 1400 of some embodiments for setting thevalues along the y-axis of a tonal response curve. The setting of thevalues along the y-axis completes the definition of the tonal responsecurve in some embodiments. FIG. 14 is described with respect to FIGS.15-16 which will be briefly described first and FIG. 17 which will bedescribed in context. FIGS. 15-17 together illustrate different possibleadjustments to the value of the mid-tone contrast points made by process1400. The values of some of the mid-tone contrast points in the imagesare slightly adjusted from the positions dictated by the mathematicalformulae herein. The adjustment is done for purposes of emphasizing somefeatures of the tonal response curves that might otherwise be difficultto see (e.g., the curvature of some parts of the curves).

FIG. 15 illustrates an adjustment of a lower mid-tone contrast point ofa tonal response curve of some embodiments. The figure shows the tonalresponse curve at an intermediate stage of setting the lower mid-tonecontrast point and at a later stage of setting the lower mid-tonecontrast point. FIG. 15 includes tonal response curve 1510 (theintermediate stage) and tonal response curve 1520 (the later stage).Tonal response curve 1510 includes black point 1512, white point 1514,lower mid-tone contrast point 1516, upper mid-tone contrast point 1518,and median point 1519. Tonal response curve 1520 includes black point1512, white point 1514, lower mid-tone contrast point 1526, uppermid-tone contrast point 1518, and median point 1519. Baselines 1515 areoverlain on the curves 1510 and 1520 to clarify the difference betweenmid-tone contrast points that are aligned with the black points andwhite points and those that are not aligned with the black points andwhite points. In the illustrated embodiment, the black point 1512, whitepoint 1514, upper mid-tone contrast point 1518, and the median point1519 are the same in both tonal response curves 1510 and 1520.

Tonal response curve 1510 and tonal response curve 1520 are visualrepresentations of mathematical functions used to remap luminance valuesof pixels of an input image into luminance values of an output image.Black point 1512 defines, for curve 1510, the maximum luminance level apixel in the input image can have in order to be remapped to a zeroluminance value (e.g., darkest possible black) in the output image.White point 1514 defines, for curve 1510, the minimum luminance level apixel in the input image can have in order to be remapped to a oneluminance value (e.g., brightest possible white) in the output image.Lower mid-tone contrast point 1516 defines the shape of the curve 1510between the black point 1512 and the median point 1519. Upper mid-tonecontrast point 1518 defines the shape of the curve 1510 between themedian point 1519, and the white point 1514.

For curve 1520, black point 1512 defines the maximum luminance level apixel in the input image can have in order to be remapped to a zeroluminance value (e.g., darkest possible black) in the output image.White point 1514 defines the minimum luminance level a pixel in theinput image can have in order to be remapped to a one luminance value(e.g., brightest possible white) in the output image. Lower mid-tonecontrast point 1526 defines the shape of the curve 1520 between theblack point 1512 and the median point 1519. Upper mid-tone contrastpoint 1518 defines the shape of the curve 1520 between the median point1519, and the white point 1514. The shape of the curves 1510 and 1520between the median points 1519 and the white points 1514 are identical.

The shape of the tonal response curve 1510 affects the luminance ofpixels of an output image that is remapped according to the curve.Luminance contrast (sometimes just called “contrast”) between two pixelsis the difference between their respective luminance values. If aremapping curve increases the difference in luminance between twopixels, that curve can be said to have increased the contrast betweenthe pixels. If a remapping curve decreases the difference in luminancebetween two pixels, that curve can be said to have decreased thecontrast between the pixels. The factor that determines whether aremapping curve will increase or decrease the contrast between twopixels of close initial luminance values is the average slope of thetonal response curve between the input points. If the curve at thatluminance value has an average slope greater than one, then the contrastis increased. If the curve has an average slope less than one then thecontrast is decreased. If the curve has an average slope equal to one,then the contrast is unchanged. The curve 1510 has a variable slope.Therefore, pixels with different luminance levels in the input imagehave different contrast adjustments when being remapped to the outputimage.

In FIG. 15, the black point 1512 and white point 1514 are closer to eachother along the x-axis than along the y-axis, so if the curve 1510 werea straight line between the two end points, then the slope would beconstant and greater than one, increasing the contrast between any twopixels with luminances between the black point 1512 and the white point1514. This hypothetical line can be called a baseline (e.g., baseline1515).

The median point 1519 is aligned with the end points. That is, themedian point 1519 lies on the baseline 1515 connecting the black point1512 and white point 1514. The lower mid-tone contrast point 1516 liesabove the baseline 1515. The lower mid-tone contrast point 1516 changesthe shape of the curve between the black point 1512 and the median point1519 by pulling up the curve. This increases the slope of the curve (andthus increases the contrast enhancement) between the black point 1512and the lower mid-tone contrast point 1516. Pulling up the curve alsodecreases the slope of the curve (and thus decreases the contrastenhancement) between the lower mid-tone contrast point 1516 and themedian point 1519. If the slope drops below one, then the contrast willbe decreased between a pair of pixels with luminance values on that partof the curve 1510 rather than increased.

The upper mid-tone contrast point 1518 also lies above the baseline1515. The upper mid-tone contrast point 1518 changes the shape of thecurve between the median point 1519 and the white point 1514 by pullingup the curve. This increases the slope of the curve (and thus increasesthe contrast enhancement) between the median point 1519 and the uppermid-tone contrast point 1518. Pulling up the curve also decreases theslope of the curve (and thus decreases the contrast enhancement) betweenthe upper mid-tone contrast point 1518 and the white point 1514. If theslope drops below one, then the contrast will be decreased between apair of pixels with luminance values on that part of the curve 1510rather than increased.

The upper portion of the curve 1520 is the same as the upper portion ofthe curve 1510. However, for the lower portion, the lower mid-tonecontrast point 1526 lies on the baseline 1515. This results in astraight line between the black point 1512 and the median point 1519 ofcurve 1520. The straight line has a constant slope greater than one soit increases the contrast of all pixels with luminance between the blackpoint 1512 and the median point 1519.

Having a straight line between the black point and the median point canbe preferable to having the lower half of the curve pulled up becauseimages generally look better with higher contrast enhancement betweenthe mid-tone contrast points and the median point and lower (or thesame) contrast enhancement between the mid-tone points and their closestend points. Making the lower portion of the line straight instead ofpulled up increases (compared to the pulled up curve) the contrastenhancement between the lower mid-tone contrast point and the medianpoint. Making the lower portion of the line straight instead of pulledup also decreases the contrast enhancement between the black point andthe lower mid-tone contrast point.

FIG. 16 illustrates an adjustment of an upper mid-tone contrast point ofa tonal response curve of some embodiments. The figure shows the tonalresponse curve at an intermediate stage of setting the upper mid-tonecontrast point and at a later stage of setting the upper mid-tonecontrast point. FIG. 16 includes tonal response curve 1610 (theintermediate stage) and tonal response curve 1620 (the later stage).Tonal response curve 1610 includes black point 1612, white point 1614,lower mid-tone contrast point 1616, upper mid-tone contrast point 1618,and median point 1619. Tonal response curve 1620 includes black point1612, white point 1614, lower mid-tone contrast point 1616, uppermid-tone contrast point 1628, and median point 1619. Baselines 1615 areoverlain on the curves 1610 and 1620, as described for the baselines1515 of FIG. 15, the baselines 1615 clarify when a mid-tone contrastpoint is aligned with the black point 1612 and the white point 1614. Inthe illustrated embodiment, the black point 1612, white point 1614,lower mid-tone contrast point 1616, and the median point 1619 are thesame in both tonal response curves 1610 and 1620.

Tonal response curve 1610 and tonal response curve 1620 are visualrepresentations of mathematical functions used to remap luminance valuesof pixels of an input image into luminance values of an output image.Black point 1612 defines, for curve 1610, the maximum luminance level apixel in the input image can have in order to be remapped to a zeroluminance value (e.g., darkest possible black) in the output image.White point 1614 defines, for curve 1610, the minimum luminance level apixel in the input image can have in order to be remapped to a oneluminance value (e.g., brightest possible white) in the output image.Lower mid-tone contrast point 1616 defines the shape of the curve 1610between the black point 1612 and the median point 1619. Upper mid-tonecontrast point 1618 defines the shape of the curve 1610 between themedian point 1619, and the white point 1614.

For curve 1620, black point 1612 defines the maximum luminance level apixel in the input image can have in order to be remapped to a zeroluminance value (e.g., darkest possible black) in the output image.White point 1614 defines the minimum luminance level a pixel in theinput image can have in order to be remapped to a one luminance value(e.g., brightest possible white) in the output image. Lower mid-tonecontrast point 1616 defines the shape of the curve 1620 between theblack point 1612 and the median point 1619. Upper mid-tone contrastpoint 1628 defines the shape of the curve 1620 between the median point1619, and the white point 1614. The shape of the curves 1610 and 1620between the black points 1612 and the median points 1619 are identical.

The shape of the tonal response curve 1610 affects the luminance ofpixels in an output image that is remapped from pixels in an input imageaccording to the curve. Because the curve 1610 has a variable slope,pixels with different luminance levels in the input image have differentcontrast adjustments when being remapped to the output image.

The black point 1612 and white point 1614 are closer to each other alongthe x-axis than along the y-axis, so if the curve 1610 were a straightline between the two end points, then the slope would be constant andgreater than one, increasing the contrast between any two pixels withluminances between the black point 1612 and the white point 1614. Theblack point 1612 and the white point 1614 are connected in the figure bythe straight line baseline 1615, which represents the hypothetical shapeof a curve comprising a straight line from black point 1612 to whitepoint 1614.

The median point 1619 is aligned with the end points. That is, themedian point 1619 lies on the baseline connecting the black point 1612and white point 1614. The lower mid-tone contrast point 1616 lies belowthe baseline. The lower mid-tone contrast point 1616 changes the shapeof the curve between the black point 1612 and the median point 1619 bypulling down the curve. This decreases the slope of the curve (and thusdecreases the contrast enhancement) between the black point 1612 and thelower mid-tone contrast point 1616. If the slope drops below one, thenthe contrast between two pixels with luminance values near that part ofthe curve 1610 will be reduced rather than increased. Pulling down thecurve also increases the slope of the curve (and thus increases thecontrast enhancement) between the lower mid-tone contrast point 1616 andthe median point 1619.

The upper mid-tone contrast point 1618 also lies below the baseline. Theupper mid-tone contrast point 1618 changes the shape of the curvebetween the median point 1619 and the white point 1614 by pulling downthe curve. This decreases the slope of the curve (and thus decreases thecontrast enhancement) between the median point 1619 and the uppermid-tone contrast point 1618. If the slope drops below one, then thecontrast between two pixels with luminance around that part of the curve1610 will be reduced rather than enhanced. Pulling down the curve alsoincreases the slope of the curve (and thus increases the contrastenhancement) between the upper mid-tone contrast point 1618 and thewhite point 1614.

The lower portion of the curve 1620 is the same as the lower portion ofthe curve 1610. However, for the upper portion, the upper mid-tonecontrast point 1628 lies on the baseline. This results in a straightline between the median point 1619 and the white point 1614 of curve1620. The straight line has a constant slope greater than one so itincreases the contrast of all pixels with luminance between the medianpoint 1619 and the white point 1614.

Having a straight line between the median point and the white point canbe preferable to having the upper half of the curve pulled down becauseimages generally look better with higher contrast increases between themid-tone contrast points and the median point and lower contrastincreases between the mid-tone points and their closest end points.Making the upper portion of the line straight instead of pulled downincreases the contrast enhancement between the median point and theupper mid-tone contrast point compared to what it would be if it werepulled down. Making the upper portion of the line straight instead ofpulled down also decreases the contrast enhancement between the uppermid-tone contrast point and the white point compared to what it would beif it were pulled down.

As mentioned above, the value along the y-axis of the black point isautomatically set to zero and the value along the y-axis of the whitepoint is automatically set to one. The process 1400 in FIG. 14 sets thevalues along the y-axis of the other three points that define the tonalresponse curve (e.g., lower mid-tone contrast point 1526, upper mid-tonecontrast point 1518, and median point 1519 of FIG. 15) in someembodiments.

Process 1400 begins by setting (at 1405) the value of the median pointto lie on the baseline connecting the black point and white point of thecurve. As shown in FIG. 15, it aligns the median point 1519 with thebaseline 1515. In some embodiments, this is done using equations (11A)and (11B).slope=1/(black_point−white_point)  (11A)median_value=slope*(median_location−black_point)  (11B)

In equations (11A) and (11B), for curve 1510 the “slope” is the slope ofthe baseline 1515, “median_location” is the location along the x-axis ofthe median point 1519 (e.g., the 50 percentile point on the histogram ofthe input image), “black_point” is the location along the x-axis of theblack point 1512, “white_point” is the location along the x-axis of thewhite point 1514.

After setting the value of the median point location, the process 1400then determines (at 1410) a value for the lower mid-tone contrast point.In some embodiments, the determined value is the average of the value ofthe median point 1519 and the black point 1512 (which as mentionedabove, is automatically set to zero). In some embodiments equation (12)is used to calculate the value of the lower mid-tone contrast point1516.LMCV=0.5*(black_value+median_value)  (12)

In equation (12) “black_value” is the value of the black point (e.g.,zero), “median_value” is the median value as calculated above, “LMCV” isthe lower mid-tone contrast point value. The calculated LMCV value ishalfway between the values of the black point and the median point. Thepreviously set location of the lower mid-tone contrast point, combinedwith the newly set value places the point either to the left of (above)the baseline or to the right of (below) the baseline.

In the case of curve 1510, illustrated in FIG. 15, the location andvalue of the lower mid-tone contrast point 1516 place it above thebaseline 1515. As mentioned above, pulling the curve up at the lowermid-tone contrast point 1516 reduces the contrast enhancement of thecurve between the lower mid-tone contrast point 1516 and the medianpoint 1519. Some embodiments prevent such reductions in mid-tonecontrast enhancement. That is, the image editing applications of someembodiments prevent the reduction of contrast enhancement between thelower mid-tone contrast point and the median point. Therefore, theprocess 1400 of some embodiments determines (at 1415) whether the valueof the lower mid-tone contrast point (e.g., lower mid-tone contrastpoint 1516) places it above the baseline (e.g., baseline 1515). If theprocess 1400 determines (at 1415) that the value of the lower mid-tonecontrast point places it above the baseline then the process 1400adjusts (at 1420) the value to put the lower mid-tone contrast point onthe baseline. This is illustrated in FIG. 15 by the adjustment of lowermid-tone contrast point 1516 in curve 1510 to a lower position as lowermid-tone contrast point 1526, on the baseline 1515 in curve 1520.

If the process 1400 determines (at 1415) that the lower mid-tonecontrast point is below the baseline, then it does not move the lowermid-tone contrast point to the baseline. This is demonstrated in FIG.16, in which the lower mid-tone contrast point 1616 of curve 1610remains in place as lower mid-tone contrast point 1616 of curve 1620rather than moving up to baseline 1615. However, FIGS. 15 and 16 do notillustrate the entire adjustment process 1400 of some embodiments. Insome embodiments, the process 1400 adjusts (at 1425) a lower mid-tonecontrast point with a value that places it below the baseline, closer tothe baseline. This will be illustrated with respect to FIG. 17,described below. The process 1400 of some embodiments takes a weightedaverage of the value of the lower mid-tone contrast point and the valueof the baseline at the location along the x-axis of the lower mid-tonecontrast point. Some embodiments perform this weighted average usingequations (13A)-(13B).LC=min(1,histogram_black+(1−histo_white))  (13A)w=CP*max(0,1-2*LC)  (13B)ALMCV=w*LMCV+(1−w)*slope*(LMCP−black_point)  (13C)

In equations (13A)-(13C) “histo_white” is the originally calculatedlocation of the white point from the histogram of the input image (e.g.,the position along the x-axis of white point 1314 from curve 1310 inFIG. 13). “histogram_black” is the originally calculated location of theblack point from the histogram of the input image (e.g., the positionalong the x-axis of black point 1312 from curve 1310 in FIG. 13). “LC”is a placeholder variable related to the distance (along the x-axis)between histogram_black and histo_white. “CP” is a curve percentage thatin some embodiments is set by the makers of the image editing andorganizing application. “LMCV” is the previously calculated lowermid-tone contrast point value (along the y-axis). “ALMCV” is theadjusted lower mid-tone contrast point value (along the y-axis). “Slope”is the slope of the baseline as calculated in equation (11A). “LMCP” isthe location of the lower mid-tone contrast point (along the x-axis).“Black_point” is the location of the black point (along the x-axis). “w”is the weighting factor for the weighted average of the lower mid-tonecontrast point and the value of the baseline at the location of thelower mid-tone contrast point. The “slope*(LMCP−black_point)” termrepresents value along the y-axis of the baseline at the location (alongthe x-axis) of the lower mid-tone contrast point. In some embodiments,the curve percentage (CP) is between 0.5 and 0.7. In some embodiments,the CP is 0.6.

As mentioned above, in some embodiments, the process 1400 adjusts (at1425) a lower mid-tone contrast point with a value that places it belowthe baseline, closer to the baseline. In some embodiments, a similaroperation (at 1445, described below) is performed on an upper mid-tonecontrast point that is above the baseline, to bring it closer to thebaseline, as described below, FIG. 17 illustrates both such operations.

FIG. 17 illustrates an adjustment of mid-tone contrast points closer toa baseline. The figure includes tonal response curves 1710 and 1720, andbaselines 1715. Curve 1710 includes lower mid-tone contrast point 1716,and upper mid-tone contrast point 1718. Curve 1720 includes lowermid-tone contrast point 1726, and upper mid-tone contrast point 1728.Curve 1710 is a tonal response curve calculated with lower mid-tonecontrast point 1716 value as determined by equation (12), above andupper mid-tone contrast point 1718 value as determined by equation (14),below. Curve 1720 is a tonal response curve recalculated with lowermid-tone contrast point 1726 value as adjusted by equation (13C), aboveand upper mid-tone contrast point 1728 as adjusted by equation (15),below. In the illustrated embodiment, a curve percentage factor (CP) of0.6 was used. Once the weighted average of the calculated lower mid-tonecontrast point 1716 is taken, the lower mid-tone contrast point 1726moves up, closer to the baseline 1715. Similarly, once the weightedaverage of the calculated upper mid-tone contrast point 1718 is taken,the upper mid-tone contrast point 1728 moves down, closer to thebaseline 1715.

After adjusting (at 1425) the value of the lower mid-tone contrast pointcloser to the baseline, the process 1400 then determines (at 1430) aninitial value for the upper mid-tone contrast point. In someembodiments, the determined value is the average of the value of themedian point 1619 and the white point 1614 (which was automatically setto one, as mentioned above). In some embodiments equation (14) is usedto calculate the value of the upper mid-tone contrast point 1618.UMCV=0.5*(white_value+median_value)  (14)

In equation (14) “white_value” is the value of the white point (e.g.,one), “median_value” is the median value as calculated above, “UMCV” isthe upper mid-tone contrast point value. The calculated UMCV value ishalfway between the values of the median point and the white point. Thepreviously set location (along the x-axis) of the upper mid-tonecontrast point, combined with the newly set value (along the y-axis)places the point either to the left of (above) the baseline or to theright of (below) the baseline.

In the case of curve 1610, illustrated in FIG. 16, the location andvalue of the upper mid-tone contrast point 1618 place it below thebaseline 1615. As mentioned above, pulling the curve down at the uppermid-tone contrast point 1618 reduces the contrast enhancement of thecurve between the median point 1619 and the upper mid-tone contrastpoint 1618. Some embodiments prevent such reductions in contrastenhancement. That is, the image editing applications of some embodimentsprevent the reduction of contrast enhancement between the median pointand the upper mid-tone contrast point. Therefore, the process 1400 ofsome embodiments determines (at 1435) whether the value of the uppermid-tone contrast point (e.g., upper mid-tone contrast point 1618)places it below the baseline (e.g., baseline 1615). If the process 1400determines (at 1435) that the value of the upper mid-tone contrast pointplaces it below the baseline then the process 1400 adjusts (at 1440) thevalue to put the upper mid-tone contrast point on the baseline. This isillustrated in FIG. 16 by the adjustment of upper mid-tone contrastpoint 1618 in curve 1610 to a higher position as upper mid-tone contrastpoint 1628, on the baseline 1615 in curve 1620.

If the process 1400 determines (at 1435) that the upper mid-tonecontrast point is above the baseline, then it does not move the uppermid-tone contrast point to the baseline. This is demonstrated in FIG.15, in which the upper mid-tone contrast point 1518 of curve 1510remains in place as upper mid-tone contrast point 1518 of curve 1520rather than moving down to baseline 1515. However, as shown in FIG. 17,the process 1400 does adjust (at 1445) an upper mid-tone contrast pointwith a value that places it above the baseline, closer to the baseline.The process 1400 of some embodiments takes a weighted average of thevalue of the upper mid-tone contrast point and the value of the baselineat the location (along the x-axis) of the upper mid-tone contrast point.Some embodiments perform this weighted average using equations(13A)-(13B) and (15).LC=min(1,histogram_black+(1−histo_white))  (13A)w=CP*max(0,1-2*LC)  (13B)AUMCV=w*UMCV+(1−w)*slope*(UMCP−black_point)  (15)

Equations (13A) and (13B) are the same equations previously described inrelation to the lower mid-tone contrast value, they are repeated herefor convenience. In equations (13A)-(13B) and (15) “histo_white” is theoriginally calculated location of the white point from the histogram ofthe input image (e.g., the position along the x-axis of white point 1314from curve 1310). “histogram_black” is the originally calculatedlocation of the black point from the histogram of the input image (e.g.,the position along the x-axis of black point 1312 from curve 1310). “LC”is a placeholder variable related to the distance (along the x-axis)between histogram_black and histo_white. “CP” is a curve percentage thatin some embodiments is set by the makers of the image editing andorganizing application. “UMCV” is the previously calculated uppermid-tone contrast point value (along the y-axis). “AUMCV” is theadjusted upper mid-tone contrast point value (along the y-axis). “Slope”is the slope of the baseline as calculated in equation (11A). “UMCP” isthe location of the upper mid-tone contrast point (along the x-axis).“Black_point” is the location of the black point (along the x-axis). “w”is the weighting factor for the weighted average of the upper mid-tonecontrast point and the value of the baseline at the location of theupper mid-tone contrast point. In some embodiments, the curve percentage(CP) is between 0.5 and 0.7. In some embodiments, the CP is 0.6.

As described above, this operation 1445 is illustrated in FIG. 17. InFIG. 17 two initially calculated mid-tone contrast points are movedcloser to the baseline. For example, in FIG. 17 initially calculatedupper mid-tone contrast point 1718 is a considerable distance above thebaseline 1715. As a result of operation 1445, the upper mid-tonecontrast point 1718 moves closer to the baseline 1715 to become uppermid-tone contrast point 1728.

While some embodiments described herein use a luminance scale from 0 to1, one of ordinary skill in the art will understand that other luminancescales are used in other embodiments. For example, some embodiments usea luminance scale from 0 to 255. Similarly, though the above describedtonal response curves operate on luminance values of input images (e.g.,in a luminance and two chrominance color space such as YIQ), someembodiments either in addition to or instead of operating on images in aYIQ space operate on images in a red, green, blue (RGB) space or othercolor spaces. Similarly, some embodiments have been described herein asbeing applied in an RGB color space. However, other embodiments performthese or similar operations in other color spaces. In some suchembodiments, the curve will be applied separately to each color.

III. Saturation/Vibrancy Enhancement

Some embodiments adjust the vibrancy of an image. That is, theembodiments make the colors present in the image more vivid. Indetermining an automatically calculated value for adjusting the vibrancyof an image, some embodiments use a histogram of the input image todetermine the vibrancy adjustment level. That is, the embodimentscalculate a histogram of an input image, then use statistics from thehistogram to determine a vibrancy setting. In some embodiments, thehistogram represents the existing saturation levels of pixels in theimage. In some embodiments, the saturation of each of the pixels in theimage is determined by the difference between the highest and lowestcolor component values of the pixel.

In order to preserve the saturation levels of images with lots offoliage or sky (e.g., images with lots of blue or green pixels), someembodiments use a lower saturation adjustment value when the image has alarge number of blue or green pixels. The application of someembodiments uses a modified histogram that counts blue and green pixelsas being more colorful (i.e., having higher saturation) than theyactually are in order to determine a lower automatic vibrancy setting.

FIG. 18 conceptually illustrates a process 1800 of some embodiments forautomatically adjusting a vibrancy value. Adjusting the vibrancy valueaffects the colors of the image (e.g., increasing vibrancy increases thesaturation of the images). In some embodiments, the vibrancy adjustmentof each pixel is done according to the pseudocode in Table 1.

TABLE 1 Vibrancy pseudocode generate skin mask to protect skin tonepixels from adjustment for non-skin tone pixels:   gray = (r + g + b) *0.33333   r = max((min(r + (r − gray) * boost), 1),0)   //boost is afunction of vibrance   g = max((min(g + (g − gray) * boost), 1),0)   b =max((min(b + (b − gray) * boost), 1),0) end

All of the values in the pseudocode of Table 1 are derivable from thevalue of the input pixel except the saturation boosting value “boost”.“boost” is a function of the vibrancy value “vibrance”. In someembodiments, the vibrancy value can be set by a user. In process 1800,the vibrancy adjustment value, “vibrance” is set automatically. Theprocess 1800 begins (at 1805) by generating a histogram of the image.The application of some embodiments sets up the bins of the histogram atthis point. The bins encompass the potential saturation values of thepixels of the image. In some embodiments, the pixels are defined bycolor components with values between 0 and 1. In such embodiments, thepotential saturation values (the possible differences between thehighest and lowest component values of a given pixel) range from 0 to 1.

The process 1800 then selects (at 1810) a pixel from the image. Thepixel can be any pixel in the image. Then the process 1800 calculates(at 1815) a saturation value for the selected pixel. The calculation ofthe saturation value of the pixel is shown in equation (16)saturation=max(r,g,b)−min(r,g,b)  (16)

In equation (16), “r”, “g”, and “b” are the red, green, and bluecomponent values of the pixel, respectively, and “saturation” is thesaturation value of the pixel. The process 1800, then determines (at1820) the color with the maximum value. If the color with the maximumvalue is determined (at 1825) to be red (i.e., not blue or green) thenthe process 1800 simply adds (at 1835) the value to the histogram (i.e.,adds one count to the bin with that saturation value). However, if thecolor is determined (at 1825) to be blue or green, then the calculatedsaturation value is adjusted before it is added to the histogram. Theapplication of some embodiments doubles the saturation value for pixelsin which the maximum color component value is either blue or green. Thecalculated saturation for such pixels is doubled before the value isadded to the histogram. One of ordinary skill in the art will understandthat the actual color component values will not be adjusted at thispoint and therefore the actual saturation value of the pixel will notchange. Only the value of the bin in the histogram to which a count isadded will change as a result of the doubling of the calculatedsaturation value.

FIG. 19 illustrates the selection of multiple pixels and how each isadded to a histogram. The figure conceptually shows the effect ofoperations 1815-1830. The figure includes input image 1900, pixels 1910,1920, 1930, and 1940, color values 1912, 1922, 1932, and 1942,saturation/histogram value 1914, saturation values 1924, 1934, and 1944,histogram values 1926, 1936, and 1948, and doubled saturation value1946. Input image 1900 conceptually illustrates an image being putthrough the automatic vibrancy enhancement process. Pixels 1910, 1920,1930, and 1940 conceptually represent pixels in an input image 1900.Color values 1912, 1922, 1932, and 1942 represent the individual colorcomponent values of pixels 1910, 1920, 1930, and 1940, respectively.Saturation values 1924, 1934, 1944, and saturation/histogram value 1914show the saturation level of each pixel 1920, 1930, 1940, and 1910,respectively. Histogram values 1926, 1936, 1948, andsaturation/histogram value 1914 show the values for saturation added tothe histogram in relation to pixels 1920, 1930, 1940, and 1910,respectively, by a histogram generation process (e.g., as described inrelation to FIG. 18). Doubled saturation value 1946, shows a value thatis twice the size of the saturation 1944 of its pixel 1940, and isrejected for inclusion in the histogram as it is over the maximum valueof the histogram (i.e., 1).

The first example of pixel 1910 in the input image 1900 has colorcomponent values 1912. The red component is the highest with a value of0.6. The blue component is the lowest with a value of 0.4. Therefore thesaturation level 1914 of pixel 1910 is 0.2. Since the maximum colorcomponent of the pixel is red, the saturation level 1914 of 0.2 is addedto the histogram. That is, one count is added to the 0.2 saturation binof the histogram. The second example of a pixel 1920 also has asaturation level 1924 of 0.2. However, this pixel has blue as thehighest value of its color component values 1922. Accordingly, theprocess 1800 doubles the saturation value to calculate a histogram value1926 of 0.4. That is, a count is added to the 0.4 bin of the histogram.Pixel 1930 has green as the highest component value so the saturationlevel 1934 of the pixel, which is 0.4 is doubled resulting in ahistogram value 1936 of 0.8. In some embodiments, the application capsthe saturation values in the histogram at 1. This is shown with respectto pixel 1940, which has a saturation value 1944 of 0.6. The highestcolor component value of pixel 1940 is green. Therefore, pixel 1940yields a double saturation value 1946 of 1.2. The double saturationvalue 1946 is then clamped to a value of 1 to calculate histogram value1948 of 1.

Returning to process 1800 (of FIG. 18), the process 1800 then determines(at 1840) whether there are more pixels that haven't been added to thehistogram. If there are more pixels, then the process 1800 returns tooperation 1810. When all the pixels have been accounted for, the process1800 identifies (at 1845) the location on the modified histogram of aspecific percentile (e.g., the 90^(th) percentile). FIG. 20 illustratesa modified histogram used for calculating a percentile of luminancevalues on the histogram. The figure has two histograms made from thesame data with the exception that one of them accounts for the green andblue pixels in the manner described above (i.e., doubling theirsaturation levels and capping the doubled levels at 1). The figureincludes histograms 2010 and 2020. Histogram 2010 has 90^(th) percentilepoint 2012 and histogram 2020 has 90^(th) percentile point 2022.Histogram 2010 is an unmodified histogram of the saturation values of animage. Histogram 2020 is a histogram of the saturation values with thesaturation values of the blue and green dominated pixels doubled beforebeing added to the histogram. The 90^(th) percentile point 2012 is thepoint at which 90% of the histogram's 2010 count lays. The 90^(th)percentile point 2022 is the point at which 90% of the histogram's 2020count lays.

The unmodified histogram 2010 trails off down to zero near the fullscale point and near the zero point. Not all images will generatehistograms that trail to zero at the ends of the scales, but the imageused to generate this histogram happens to have no fully saturatedpixels and no pixels with zero saturation (completely gray, white, orblack). The modified histogram 2020 has lower saturation level pixels(blue and green ones) counted near the full scale point, so it doesn'ttrail off to zero near the full scale point. Any blue or green pixelswith a saturation level at or above 0.5 are added to the histogram asthough they had saturation levels of 1, therefore there is a spike atthe top of the scale (i.e., 1 in this example) in the modified histogram2020. The additional counts near the top end of the scale move the90^(th) percentile point 2022 to a higher value (along the x-axis) thanthe 90^(th) percentile point 2012. For some images, the additionalcounts move the 90^(th) percentile point all the way to the top of thescale (e.g., if more than 10% of the pixels have blue or green maximumcomponent values and saturation values at or above 0.5). However for theimage used to generate histograms 2010 and 2020, the additional countson the high end move the 90^(th) percentile point from about 0.7 (point2012) to about 0.8 (point 2022).

Once the 90^(th) percentile point is calculated, the process 1800 sets(at 1850) a mathematical formula shown as equation (17) to determine asetting for vibrancy.vibrance=8*0.4*((1−percentile_location)^3)*(percentile_location^1.6)  (17)

In equation (17), “vibrance” is the automatically determined vibrancysetting. “percentile_location” is the location of the specifiedpercentile (e.g., 90^(th) percentile). FIG. 21 illustrates a graph ofpercentile location versus automatic vibrancy settings. The graphincludes an illustration of the effects of using an adjusted histogram(i.e., histogram 2020 in FIG. 20) versus using an unadjusted histogram(i.e., histogram 2010 in FIG. 20). The figure includes graph 2100,vibrancy setting curve 2110, unadjusted mark 2120, and adjusted mark2130. The graph 2100 has an x-axis with the locations on it matching thelocations of bins of the histogram and a y-axis with vibrancy settingvalues on it. The vibrancy setting curve 2110 correlates values on thex-axis with values on the y-axis. The unadjusted mark represents thelocation of the 90^(th) percentile on unadjusted histogram 2010 in FIG.20. The adjusted mark 2130 represents the location of the 90^(th)percentile on adjusted histogram 2020 in FIG. 20.

In the graph 2100, the x-axis represents a specified percentile of thehistogram (here, the 90^(th) percentile). The y-axis represents vibrancysettings to be correlated with calculated 90^(th) percentile points onthe x-axis. As shown on the graph 2100, the adjusted mark 2130correlates with a smaller automatically set vibrancy value than theunadjusted mark 2120. Accordingly, the modification of the histogram hasproduced a lower vibrancy setting than would be the case for anunmodified histogram.

Some embodiments use the following sets of equations (18A)-(21C) toconvert the automatically derived “vibrance” setting into new colorcomponent values for a given pixel. The individual sets of equationswill be explained between listings.r ₁=max(min(R ₁,0.9999),0.0001)  (18A)g ₁=max(min(G ₁,0.9999),0.0001)  (18B)b ₁=max(min(B ₁,0.9999),0.0001)  (18C)rdelta=R ₁ −r ₁  (18D)gdelta=G ₁ −g ₁  (18E)bdelta=B ₁ −b ₁  (18F)

Equations (18A)-(18F) remove high dynamic range data (e.g., colorcomponent data above 0.9999, or below 0.0001) from the red (R₁), green(G₁) and blue (B₁) components of the pixel to calculate temporary red(r₁), green (g₁) and blue (b₁) components and store the overage as deltavalues (rdelta, gdelta, and bdelta respectively). The delta values willbe added back to the pixel after the boost phase. For example, if apixel has a red (R₁) value of 1.5, the rdelta of 0.5001 will be storedand the pixel value will be represented in the vibrancy boost by a red(r₁) value of 0.9999, after the boost phase, the rdelta 0.5001 will beadded back into the red value.gray=(r ₁ +g ₁ +b ₁)*0.33333  (19A)gi=1.0/gray  (19B)gii=1.0/(1.0−gray)  (19C)rsat=min(max((r ₁−gray)*gii,(gray−r ₁)*gi),0.99999)  (19D)gsat=min(max((g ₁−gray)*gii,(gray−g ₁)*gi),0.99999)  (19E)bsat=min(max((b ₁−gray)*gii,(gray−b ₁)*gi),0.99999)  (19F)sat=max(rsat,gsat,bsat)  (19G)skin=(min(max(0,min(r ₁ −g ₁ ,g ₁*2−b₁))*4*(1−rsat)*gi,1))*0.7+0.15  (19H)tsat=1−(1−sat)^(1+3*vibrance)  (19I)boost=(1−skin)*(tsat/sat−1)  (19J)

Equations (19A)-(19J) are used to compute saturation adjustment value“boost” for the pixel. “Gray” represents an average of the colorcomponents and is used to compute saturation adjustments (“rsat”,“gsat”, “bsat”) for each color component. The largest of thesesaturation adjustments is then used as an overall saturation adjustmentvariable “sat”. Equation (19H) is used to create a term to protect theskin tone pixels from being boosted. In some embodiments, the skincolors were previously adjusted by the white balance operation andchanging them further by too large an amount is not desirable. The“vibrance” term previously calculated in equation (17) is used inequation (19I) to affect the “tsat” value, which is used for calculatingthe saturation boost (“boost”) value in equation (19J). As equation(19J) shows, the “boost” value is tempered by the “skin” value. When the“skin” value is relatively large, indicating skin tone, the boost valueis reduced to a fraction of what it would be for a non skin tone pixel.In some embodiments, the equations are adjusted so that the “Boost”value is zero for skin tone pixels. In some embodiments the code isadjusted so that the “Boost” value is otherwise not applied to skin tonepixels.r ₂=max((min(r ₁+(r ₁−gray)*boost),1),0)  (20A)g ₂=max((min(g ₁+(g ₁−gray)*boost),1),0)  (20B)b ₂=max((min(b ₁+(b ₁−gray)*boost),1),0)  (20C)

Equations (20A)-(20C) calculate the adjustment to the individual colorcomponents from the saturation boost term “boost”. The equationsgenerate pixel component values r₂, g₂, and b₂ by subtracting theaverage pixel value from the component value (r₂, g₂, and b₂,respectively), boosting the remaining portion of the component value bya factor of “boost” and then adding the component value to the boostedvalue. For example, for pixel 1920 (in FIG. 19) with its r (0.4), g(0.5), and b (0.6) values, the average value (i.e. the “gray” number)would be 0.5. Assuming a boost value of 2 (this is not the calculatedvalue, it is used here for clarity), the blue value (0.1 above the“gray” level) would increase by 0.2 (from 0.6 to 0.8). The green value(at the “gray” level) would be unchanged. The red value (0.1 below the“gray” level) would drop by 0.2 (from 0.4 to 0.2).

For component values that were originally between 0 and 1 (or in someembodiments, between 0.0001 and 0.9999, inclusive), the adjustment stopshere. However, for pixels with high dynamic range values, previouslystored as deltas by equations (18D)-(18F), the deltas are added back byequations (21A)-(21C).R ₂ =r ₂ +rdelta  (21A)G ₂ =g ₂ +gdelta  (21B)B ₂ =b ₂ +bdelta  (21C)

After adjusting (at 1850 of FIG. 18) the vibrancy of the image, theprocess of automatically setting the vibrancy ends and in someembodiments, the process of setting up a tonal response curve starts.

IV. Shadow Lift Enhancement

After performing the saturation/vibrancy enhancement described above,the image editing applications of some embodiments perform a shadow liftoperation. A shadow lift operation increases the contrast of the darkareas of the image. Some embodiments use a variable gamma adjustment toperform the shadow lift operation. A description of the shadow liftoperation of some embodiments can be found in U.S. patent applicationSer. No. 13/152,811, now issued as U.S. Pat. No. 8,754,902, entitled“Color-Space Selective Darkness And Lightness Adjustment”, which isincorporated herein by reference. Proper shadow lifting improves theappearance of a digital image by allowing items in the shadows to beseen more clearly. However, lifting the shadow image too much can makethe image look worse. Therefore, the automatic enhancement system ofsome embodiments generates a structure histogram of the image and usesit to determine what level of shadow lift to apply to the image.

A. Structure Histograms

A traditional histogram of an image (e.g., a luminance histogram)determines statistics about the individual pixels of image. In the caseof a luminance histogram, the luminance values of the image are groupedinto bins for the histogram, with each bin containing a range ofluminance values. The established range for a bin in the histogram couldbe simply one increment of luminance (i.e., the smallest possibledifference in luminance available on the applicable scale), meaning thateach bin contains pixels with identical luminance values. Once the binsare established, the histogram is generated by placing one count in abin for each pixel with a value in the established range for the bin. Tovisually display the histogram, a graph can be generated with, forexample, the bin values on the x-axis and the number of pixels in eachbin on the y-axis. One limitation of such a traditional histogram isthat it does not contain any information about the structure of theimage, only about the individual pixels. Therefore, two images withcompletely different scenes can result in identical histograms. Astructure histogram, on the other hand, does contain information aboutthe structure of the image.

FIG. 22 illustrates the differences between structure histograms of someembodiments and traditional histograms for two different images. Thefigure includes image 2210, conventional histogram 2212, with peaks 2213and 2214, structure histogram 2215, with peaks 2216 and 2217, image2220, conventional histogram 2222, with peaks 2223 and 2224, andstructure histogram 2225, with peak 2226.

Image 2210 is the image of a baby with a bonnet. Conventional histogram2212 is a conventional histogram of the image 2210 of the baby. Thehistogram has peaks 2213 and 2214. The peaks 2213 and 2214 are measuresof large counts in the conventional histogram. Structure histogram 2215is a structure histogram of the image 2210 of the baby. The structurehistogram 2215 has peaks 2216 and 2217. The peaks 2216 and 2217 aremeasures of large counts of pixels with neighbors in that range in thestructure histogram. Image 2220 is an image generated from the image2210 of a baby with a bonnet. In the image 2220, the pixels of the imageof the baby have been rearranged in order of luminance. The brightestpixels from the baby picture are on the bottom of image 2220 and thedarkest pixels are on the top of the image. Conventional histogram 2222is a conventional histogram of the image 2220 of the ordered pixels. Thehistogram has peaks 2223 and 2224. The peaks 2223 and 2224 are measuresof large counts in the conventional histogram. Structure histogram 2225is a structure histogram of the image 2220 of the ordered pixels. Thestructure histogram 2225 has a peak 2226. The peak 2226 is a measure ofa large count in the structure histogram. Pseudocode for generating theconventional histograms 2212 and 2222 is found in Table 2.

TABLE 2 Conventional Histogram Pseudocode int histogram[256] = 0 //define a histogram having 256 elements or bins int count = 0 // count isa measure of the number of pixels in the image for p in image // foreach pixel in an image do ... histogram[luminance(p)] =histogram[luminance(p)] +1 count = count +1 end “p” (pixel) loop plothistogram/count

In Table 2. “luminance(p)” is a function that returns the luminance ofthe pixel (p) (e.g., an integer between 0 and 255).

The pseudocode of Table 2 generates a histogram whose values are basedonly on the luminance values of the individual pixels in the image. Whenthe pseudocode in table 2 is applied to the baby image 2210, it producesconventional histogram 2212. When the pseudocode in Table 2 is appliedto the image 2220 of ordered pixels, it produces conventional histogram2222. Because the individual pixels in image 2210 have the same valuesas the individual pixels in image 2220, the conventional histograms 2212and 2222 generated from the respective images are identical. The peaks2213 and 2223 indicate that there are a large number of very brightpixels in each image (e.g., the baby's clothes in image 2210 and thepixels near the bottom of image 2220). The peaks 2214 and 2224 indicatethat there are a large number of very dark pixels in each image (e.g.,the background of the baby in image 2210 and the pixels near the top inimage 2220).

Even though the images 2210 and 2220 are very different from each other,there is no difference in the conventional histograms 2212 and 2222.However, a shadow lift setting that would be a good setting for image2210, would not necessarily be a good setting for shadow lifting anotherimage with the same pixels in a different order, such as image 2220.Therefore the conventional histograms are not a useful tool for beingthe sole determining factor of what is a good shadow lift setting foreach of the two images. The structure histograms 2215 and 2225 aredifferent for the two different images 2210 and 2220. Pseudocode forgenerating the structure histograms is found in Table 3.

TABLE 3 Illustrative structure histogram pseudocode int histogram[256] =0 // define a histogram having 256 elements or bins int count = 0 //count is a measure of the complexity of the image int i = 0 for p inimage // for each pixel in an image do ... for n of p // for eachneighbor pixel do ... // increment the value of each histogram element// between p and its neighbor pixel n for i = { min( lum(p), lum(n) ) +1} to { max(lum(p), lum(n)) − 1 } histogram [i ] = histogram [i ] + 1count = count + 1 end “i” loop // bins between p and n pixels end “n”(neighbor) loop end “p” (pixel) loop plot histogram/count

In Table 3, lum(p) is a function that returns the luminance of pixel (p)(e.g., an integer between 0 and 255). The pseudocode for the structurehistogram is a bit more complicated than the pseudocode for theconventional histogram. Like the conventional histogram pseudocode, thestructure histogram pseudocode checks each pixel in the image. However,where the conventional histogram values are determined by the luminancevalues of the individual pixels alone, the structure histogram valuesare determined by the relationship between each pixel and itsneighboring pixels. For example, if a pixel has a luminance of 200, andeach of its eight neighbors has a value of 205, then the histogram willadd one to each of the bins between the pixel (200) and its neighbors(205). That is, one count will be added to each bin 201, 202, 203, and204 for each of the neighbors. This will add a total of 32 to thevariable “count” which is used to normalize the histogram once it iscomplete. In some embodiments, neighboring pixels with the same valueand/or pixels differing from each other by one value are not counted inthe structure histogram.

A bin in a structure histogram generated by the pseudocode of Table 3only gets a count if there are neighboring pixels with differing values.In image 2210, there is a great deal of fine structure (texture) in thedark background of the image. Accordingly, the pseudocode for thestructure histogram includes a peak 2217 at a fairly low bin level. Incontrast, the ordered pixels in image 2220 have very little structure,the black rectangle at the top includes many rows of pixels with thesame value (e.g., zero), then a one row transition to pixels of anothervalue. If the pixels in the next row are only one value away from theprevious row (e.g., the upper row is zero and the next row is one) someembodiments will not add anything to any bin as a consequence. Even inembodiments which count pixels that differ by one value, there arerelatively few neighboring dark pixels with differing values in image2220. Therefore, there is no peak in structure histogram 2225 analogousto peak 2217 in structure histogram 2215.

In image 2210, there is large amount of detail and texture among thebright pixels, therefore there is another peak 2216 near the high end ofthe histogram scale. In image 2220, the amount of detail overall isseverely reduced compared to the amount of detail in image 2210. Thisresults in a lower value for the “count” variable used to normalizestructure histogram 2225 than the “count” value used to normalizestructure histogram 2215. Because the normalizing factors are different,the sizes of the peaks cannot be directly compared from one structurehistogram 2215 or 2225 to the other. Accordingly, the large peak 2226 ofstructure histogram 2225 may represent less actual detail than thesmaller peak 2216 of structure histogram 2215. However, the peak 2226does signify that what little structural detail is left in image 2220can be found among the brightest pixels.

Structural histograms, such as described above can be used by an imageediting application of some embodiments to determine a desirable settingfor an automatic shadow lift operation. Other types of structuralhistograms can also be used in some embodiments. For example, astructural histogram could be generated by a down sampled image. Or usea sample of pixels in an image to calculate a histogram rather than allthe pixels in an image. In some embodiments, fewer neighboring pixelscould be used. For example, for natural images, using a single neighbor(e.g., the lower right neighbor) for each pixel yields satisfactoryresults. Further details on structure histograms can be found in U.S.patent application Ser. No. 13/412,368, now published as U.S.2013/0229541, filed Mar. 5, 2012, which is incorporated herein byreference.

B. Automatic Setting for Shadow Lifting

FIG. 23 conceptually illustrates a process 2300 of some embodiments forautomatically generating a shadow lift enhancement input value. Theprocess 2300 begins by calculating (at 2305) a structure histogram andother histograms for the image. The structure histogram calculated forthe image in some embodiments can be any of the structural histogramsdescribed in U.S. application Ser. No. 13/412,368, now published as U.S.2013/0229541, or any other structural histogram that distinguishesbetween images with identical pixel values that are arranged indifferent arrangements.

Once a structure histogram and other histograms are calculated, theprocess 2300 determines (at 2310) various statistics of the histograms.For example, the process 2300 can determine the location of peaks in thehistograms, the location of various percentiles of the histograms (e.g.,the bins with the 5^(th) percentile, the 10^(th) percentile, the 90^(th)percentile, etc.). The process of some embodiments also calculates thewidths of the peaks, the overall complexity of the image (e.g., byexamining the “count” variable). After determining the various relevantstatistics from the histograms, the process 2300 calculates (at 2315) ashadow lift setting based on the statistics.

The image editing application of some embodiments uses an empiricallyderived formula for determining a setting for the shadow liftenhancement. The generation of such a formula will be described inrelation to FIGS. 24A, 24B, and 25, before returning to the discussionof FIG. 23. FIG. 24A conceptually illustrates the derivation of anequation for automatically determining a setting for shadow lifting. Thefigure includes image data items 2401-2408 and function box 2420. Theimage data items 2401-2408 each represents a set of statistics for aseparate image, coupled with a maximum setting. The function box 2420represents a best fit function derived from the statistics and settingsof image data items 2401-2408.

For each of the image data items 2401-2408, the statistics are structurehistogram statistics of the image represented by that data item. In someembodiments, the structure histogram algorithms and other histogramalgorithms used to generate the statistics of image data items 2401-2408are the same structural histogram algorithms and other histogramalgorithms that the application uses in operation 2305. The settings ineach of the data items 2401-8 are determined by a person who selectedthe highest setting (from multiple possible shadow lift settings) thatproduced good results for that image. That is, someone looked at animage at multiple shadow lift levels and decided what the highest shadowlift level could be before the image started looking worse withincreased shadow lift settings. The function in function box 2420 isderived from the multiple data items 2401-2408. The actual number ofimages used to generate the function can be greater than 8, equal to 8,or less than 8 in some embodiments. For example, in some embodiments,the number of analyzed images is in the hundreds.

As mentioned above, in some embodiments, the process 2300 (of FIG. 23)calculates multiple histograms. The application of some embodimentscalculates a conventional luminance histogram, a cumulative luminancehistogram that reports the percentile of any value in the luminancehistogram, and a structure histogram. The application of someembodiments extracts multiple data points from the histograms. In someembodiments, these data points include some or all of (1) maximums ofany or all histograms, (2) values of each histogram at the location ofthe structure histogram peak, (3) values of each histogram at half theheight of the structural peak to the right of the structural peak, (4)values of each histogram at half the height of the structural peak tothe left of the structural peak, (5) the sum of the histograms withineach of the following percentile bands of the structure histogram: 0-5%5-30% 30-60% 60-100%, (6) the values at 25% and 50%, (7) the sums at 25%and 50%. In some embodiments, whenever the luminance histogram value(along the y-axis) is recorded as an input, the luminance location(along the x-axis) is also recorded as an input. In addition, theapplication of some embodiments includes some or all possibledifferences and products of pairs of those numbers in the input data aswell. A function in some embodiments is generated by a mathematicalregression using the multiple images and the types of terms describedabove to generate weighting values for each of a large number of terms.In some embodiments, the regression generates a function with values forhundreds of terms and combinations of terms. For some embodimentsdescribed in this paragraph, and for some embodiments elsewhere in thisapplication “values” can include both y-axis values and x-axislocations.

FIG. 24B conceptually illustrates the use of the derived function withthe statistics of a current structure histogram and other histograms.The figure includes function box 2420 and current image structurehistogram statistics and other histogram statistics 2430. The statistics2430 of the current image are fed into the previously derived formula2420 and the formula generates an initial shadow lift setting 2440.

The initial shadow lift setting 2440 determines an amount to enhancedetail in the darker areas (shadows) of an image. However, not alldetails in shadows are real. Images can have digital artifacts. Adigital artifact is a visible object on the image that does notrepresent anything physically present in the scene from which the imagewas captured. Artifacts can mimic texture in an image taken of a smoothbackground. Any image has an associated International Organization forStandardization (ISO) value. The ISO value is a measure of thesensitivity of the camera during the capture of a particular image. Themore sensitive the camera, the larger the ISO value is. The ISO value isknown in the art and is based in part on the exposure time and thebrightness of a scene as well as the aperture f-number of the camera. Ingeneral, the larger the ISO number, the more digital artifacts will befound in the dark areas of the image. A high amount of shadow lifting ofan image with a high ISO number can enhance the artifacts and make themhighly visible (which is undesirable). Accordingly, some embodimentsreduce the calculated shadow lift setting of the artifacts by applying afunction of the ISO number.

FIG. 25 illustrates the effects of too high a shadow lift setting on animage with a high ISO. The images in the figure are both adjusted imagesof an input image with a high ISO value, one is adjusted with a lowshadow lift setting, the other is adjusted with a high shadow liftsetting. FIG. 25 includes shadow lifted images 2510 and 2520. Image 2510has been adjusted with a shadow lift operation with a low setting, toaccount for the high ISO value of the input image. Image 2520 has beenadjusted with a high shadow lift operation. The dark water in thebackground of image 2520 includes speckles that are artifacts caused bythe high ISO value rather than being an accurate depiction of the scenefrom which the input image was captured. Thus the image 2520demonstrates the advantages of keeping the shadow lift setting low forimages with a high ISO value.

Returning to FIG. 23, once the initial setting is calculated (at 2315),the process determines (at 2320) whether the ISO value of the image isknown. That is, the process determines whether the ISO value for thatparticular image is stored as metadata of the image. If the ISO value isnot stored as metadata of the image, then the process sets (at 2325) adefault value for the ISO. In the applications of some embodiments, thedefault ISO number is 100. Once the ISO value is either identified fromthe metadata, set by default (or in some embodiments identified by theuser), the process 2300 limits (at 2330) the initially determined shadowlift setting by using equation (22)Shad_(—) f=max(0.25,0.6−ISO/16000)*tanh(min(1,max(0,Shadow_initial)))  (22)

In equation (22), “Shadow_initial” is the initial shadow valuedetermined in operation 2315 from the empirically derived formula forthe shadow lift calculation. “ISO” is the ISO value used for the image,either the actual ISO taken from the metadata of the image or thedefault ISO of 100 from operation 2320. “Shad_f” is the finalautomatically calculated shadow lift setting. By application of equation(22), the initial shadow level will go to a number between 0 and about0.457. One of ordinary skill in the art will understand that otherfunctions of the ISO value can be used in some embodiments to temper theshadow lift setting. The final shadow lift setting will then be appliedon a pixel by pixel and color by color basis to the shadow adjustmentequations (23A)-(23C).r _(adjusted) =r _(input)^(2^((Shad_(—) f−(blur/colorscale))*2)  (23A)g _(adjusted) =g _(input)^(2^((Shad_(—) f−(blur/colorscale))*2)  (23B)b _(adjusted) =b _(input)^(2^((Shad_(—) f−(blur/colorscale))*2)  (23C)

In equations (23A)-(23C) “r_(adjusted)” is the red value of the pixelafter adjustment, “r_(input)” is the red value of the pixel in the inputimage, “g_(adjusted)” is the green value of the pixel after adjustment,“g_(input)” is the green value of the pixel in the input image,“b_(adjusted)” is the blue value of the pixel after adjustment,“b_(input)” is the blue value of the pixel in the input image. “Shad_f”,as in the previous equation, is the final automatically calculatedshadow value. “blur” is the value of a corresponding pixel in a Gaussianblur of the input image (the Gaussian blurred image is described furtherin subsection C, below). “Colorscale” is a scaling variable that servesto increase the colorfulness of the image and is set to various valuesin the application of various embodiments. In some embodiments it is setto 0.5. In some embodiments, “colorscale” has different values for oneor more of the color components. U.S. patent application Ser. No.13/152,811, now issued as U.S. Pat. No. 8,754,902 has more detail on theshadow lifting of input images. Once the shadow lift setting has beencalculated, the process 2300 applies (at 2335) the shadow lift to theimage.

C. Skin Protection in Shadow Lifting

U.S. patent application Ser. No. 13/152,811, now issued as U.S. Pat. No.8,754,902, describes the manual adjustment of shadow images using a userinput. The patent application describes masking of skin areas from theshadow lifting process in order to avoid desaturation of the skincolors. However, in some embodiments described herein, the protection ofskin from shadow lifting values is shut off for large values of shadowlifting, or when no faces are found in the image. FIG. 26 conceptuallyillustrates the inputs that go into generating a shadow image. Thefigure includes inputs that are always used and an input that isselectively used. The figure includes input digital image 2610 (in alinear RGB color space), a skin mask 2620, a blurred image 2630, asetting 2640, and a shadow adjusted image 2650. The input digital imagein some embodiments is the output image from the vibrancy adjustment. Insome cases the input image is any digital image to be adjusted, whetherit had been previously adjusted or not. The skin mask 2620 is a mask ofthe image that distinguishes those areas of the image with skin fromthose areas that are not skin. The blurred image 2630 is a Gaussianblurred version of the input image used to temper the gamma adjustmentof the image. The setting 2640 is either the user setting or theautomatic setting. The shadow image 2650 is the shadow lifted adjustedimage.

From an input digital image 2610, the applications of some embodimentsgenerate a skin mask 2620 that identifies regions of the imagecontaining skin. In some embodiments, the skin regions are identified bycolor. In some embodiments, the skin mask is used to designate areasthat will not have the shadow lifting operation performed on them. Theskin mask in some embodiments designates areas that will have a reducedshadow lift operation performed on them (or no shadow lift operation atall in some embodiments). For example, some embodiments generate asingle gamma function for the skin areas and a variable gamma functionfor the non-skin areas. Some such embodiments then use the skin mask tomodulate between these gamma values on a per-pixel basis. More on skinmasking can be found in U.S. patent application Ser. No. 13/152,811, nowissued as U.S. Pat No. 8,754,902. Some embodiments turn off skinprotection under some circumstances, which will be described below withrespect to FIGS. 27-29.

In order to perform the shadow lift operation, some embodiments of theapplication generate a Gaussian blurred image 2630 to use with equations(23A)-(23C), above. The Gaussian blurred image allows local gammaadjustment with the darker areas receiving more extreme adjustments andthe lighter areas receiving less extreme adjustments. In that way, theshadows get lifted but the bright areas are only changed slightly. TheGaussian image (the “blur” term in the equations) is applied in theequations (23A)-(23C) on a pixel by pixel basis, so each pixel in theimage (other than skin masked areas in some embodiments) gets its owngamma adjustment. The setting 2640 can be any setting the user chooses,or can be the automatically determined setting described in subsectionB, above. Together, the setting 2640, the digital image 2610, theGaussian blurred image 2630, and the skin mask 2620 (when applied) willdetermine a value for each pixel in the adjusted shadow image 2650.

FIG. 27 conceptually illustrates a process 2700 of some embodiments forapplying or not applying skin tone protection in a shadow liftingoperation. The process will be described in relation to FIGS. 28 and 29,which will be briefly described first. FIG. 28 illustrates an image tobe adjusted with a low shadow lift setting. The figure includes theoriginal image and a conceptual version of a shadow lifting mask. Thefigure includes input image 2810 and shadow lifting mask 2820. Theshadow lifting mask 2820 includes Gaussian blurred areas 2822, 2824, and2826, and skin tone mask area 2828. The input image 2810 is an image ofa person, and thus includes the face of the person. The shadow liftingmask 2820 is a conceptual combination of the Gaussian blurred image withthe skin tone mask. In the areas 2822-2826, there is no skin, thereforethe Gaussian blurred image is used to determine the shadow liftadjustment of the areas 2822-2826.

The Gaussian blurred areas each provide their own level of gammacorrection, which is why each of the areas (hair, shirt, and sky) isshown in the figure in a different shade of gray. One of ordinary skillin the art will understand that although the different brightness levelsof each area are shown as uniform within a large area, in a real image,the Gaussian blur creates a set of different gamma adjustments that aredifferent on a much smaller scale than a whole shirt or a whole head ofhair. One of ordinary skill in the art will understand that although theskin protection mask in some embodiments is a separate mask from theGaussian blurred image, in other embodiments, the skin protection maskand Gaussian blurred image are provided as a combined mask that bothprotects the skin tones and lifts the shadows. Similarly one of ordinaryskill in the art will understand that in some embodiments, some or allof the masks are conceptual masks and only individual pixels of themasks are calculated at any one time. However, in other embodiments someor all of the masks are calculated independently and then the calculatedmask is applied (e.g., on a pixel by pixel basis). Furthermore, in someembodiments, the skin mask for the shadows is calculated in the same wayas, or in a similar way to, the skin mask described with respect toequation (19H).

FIG. 29 illustrates an image to be adjusted with a high shadow liftsetting. The figure includes the original image and a conceptual versionof a shadow lifting mask. The figure includes input image 2910 andshadow lifting mask 2920. The shadow lifting mask includes Gaussianblurred areas 2922, 2924, and 2926, and 2928. The input image 2910 is animage of a person, and thus includes the face of the person. The shadowlifting mask 2820 is a conceptual illustration of a Gaussian blurredimage with no skin tone mask. In the areas 2922-2926, there is no skin,therefore the Gaussian blurred image is used to determine the shadowlift adjustment of the areas 2922-2926. In area 2928 there is skin, butin the illustrated embodiment, a high shadow lift setting eliminates theskin masking. The Gaussian blurred areas each provide their own level ofgamma correction, which is why each of the areas (hair, shirt, face, andsky) is a different shade of gray.

Process 2700 in FIG. 27 begins by receiving (at 2705) a shadowadjustment setting. This setting can be received from the automaticshadow lift calculator (e.g., calculator 350) or from a user setting.The process then determines (at 2710) whether there are any faces in theimage. If there are no faces, then the process adjusts (at 2715) theshadows without skin tone protection. If there are faces then theprocess 2700 determines (at 2720) whether the shadow adjustment settingis above a threshold setting. If the setting is below the thresholdsetting, then the process 2700 adjusts (at 2925) the shadows with skintone protection. This is illustrated in FIG. 28 where shadow liftingmask 2820 includes skin tone protection area 2828. If the setting isdetermined (at 2720) to be above the threshold, then the skin toneprotection is turned off as illustrated in FIG. 29 in which the shadowlifting mask 2920 does not include a skin tone protection area, butrather includes Gaussian blurred area 2928 over the face and neck of theperson shown in the image 2910.

The above described embodiments of the shadow lift operation do notadjust the black point value of the previously described tonal responsecurve. However, the shadow lift operation may have effects on the imagethat include (but are not limited to) effects similar to moving theblack point of the tonal response curve. For example, the shadow liftoperation may cause some dark pixels to darken further or to become lessdark.

While many of the figures above contain flowcharts that show aparticular order of operations, one of ordinary skill in the art willunderstand that these operations may be performed in a different orderin some embodiments. Furthermore, one of ordinary skill in the art willunderstand that the flowcharts are conceptual illustrations and that insome embodiments multiple operation may be performed in a single step.For example, in the tonal response curve flowchart of FIG. 14, in someembodiments a single mathematical equation determines both what isdescribed as the initial upper mid-tone contrast value and adjusts thesetting closer to the line. Similarly, in some embodiments a singlecombined equation determines whether to set the mid-tone point on theline and what value to give the mid-tone contrast point if it is not onthe line.

In some of the descriptions of images described herein, some datacalculations are shown as whole number (i.e., integer) calculations.Furthermore, some image formats described herein use integer values forthe images (e.g., integers from 0 to 255). However, in some embodiments,some or all data calculations and computations are made with floatingpoint values. In these cases, the image adjustments are based onfloating point computations that are more precise (e.g., no round-offloss of data within a series of cumulative calculations) thancomputations based on integer values. Accordingly, in some embodiments,image detail is preserved by treating all values (i.e., integer anddecimal values alike) as floating point values in order to perform anycalculations or computations for making image adjustments. In someembodiments, the data is returned to integer form upon saving an imagein an integer based format. Furthermore, in some embodiments theversions of the images displayed use the standard integer values fortheir color components, rounded from the floating point values storedfor the various pixels in the image data of the image editingapplication.

Just as floating point data is used to preserve factional values, someembodiments use extended range data (e.g., data above or below the scaleof the image storage format) in order to avoid losing detail that maylater be returned to the normal scale range via some other operation ofthe image editing application. In some such embodiments, the visuallypresented version of the image presents the above range data as thoughit were at the top of the allowable scale, even though the actual datais allowed to have values exceeding the top of the normal scale.

While some embodiments described herein use a luminance scale from 0 to1, one of ordinary skill in the art will understand that other luminancescales are used in other embodiments. For example, some embodiments usea luminance scale from 0 to 255. Similarly, though some of the abovedescribed enhancements were described as operating on luminance valuesof input images (e.g., in a luminance and two chrominance color spacesuch as YIQ), some embodiments either in addition to or instead ofoperating on images in a YIQ space operate on images in a red, green,blue (RGB) space or other color spaces. Similarly, some embodiments havebeen described herein as being applied in an RGB color space. However,other embodiments perform these or similar operations in other colorspaces.

V. Electronic Systems

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or morecomputational or processing unit(s) (e.g., one or more processors, coresof processors, or other processing units), they cause the processingunit(s) to perform the actions indicated in the instructions. Examplesof computer readable media include, but are not limited to, CD-ROMs,flash drives, random access memory (RAM) chips, hard drives, erasableprogrammable read-only memories (EPROMs), electrically erasableprogrammable read-only memories (EEPROMs), etc. The computer readablemedia does not include carrier waves and electronic signals passingwirelessly or over wired connections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storagewhich can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

A. Mobile Device

The image editing and viewing applications of some embodiments operateon mobile devices. FIG. 30 is an example of an architecture 3000 of sucha mobile computing device. Examples of mobile computing devices includesmartphones, tablets, laptops, etc. As shown, the mobile computingdevice 3000 includes one or more processing units 3005, a memoryinterface 3010 and a peripherals interface 3015.

The peripherals interface 3015 is coupled to various sensors andsubsystems, including a camera subsystem 3020, a wireless communicationsubsystem(s) 3025, an audio subsystem 3030, an I/O subsystem 3035, etc.The peripherals interface 3015 enables communication between theprocessing units 3005 and various peripherals. For example, anorientation sensor 3045 (e.g., a gyroscope) and an acceleration sensor3050 (e.g., an accelerometer) is coupled to the peripherals interface3015 to facilitate orientation and acceleration functions.

The camera subsystem 3020 is coupled to one or more optical sensors 3040(e.g., a charged coupled device (CCD) optical sensor, a complementarymetal-oxide-semiconductor (CMOS) optical sensor, etc.). The camerasubsystem 3020 coupled with the optical sensors 3040 facilitates camerafunctions, such as image and/or video data capturing. The wirelesscommunication subsystem 3025 serves to facilitate communicationfunctions. In some embodiments, the wireless communication subsystem3025 includes radio frequency receivers and transmitters, and opticalreceivers and transmitters (not shown in FIG. 30). These receivers andtransmitters of some embodiments are implemented to operate over one ormore communication networks such as a GSM network, a Wi-Fi network, aBluetooth network, etc. The audio subsystem 3030 is coupled to a speakerto output audio (e.g., to output different sound effects associated withdifferent image operations). Additionally, the audio subsystem 3030 iscoupled to a microphone to facilitate voice-enabled functions, such asvoice recognition, digital recording, etc.

The I/O subsystem 3035 involves the transfer between input/outputperipheral devices, such as a display, a touch screen, etc., and thedata bus of the processing units 3005 through the peripherals interface3015. The I/O subsystem 3035 includes a touch-screen controller 3055 andother input controllers 3060 to facilitate the transfer betweeninput/output peripheral devices and the data bus of the processing units3005. As shown, the touch-screen controller 3055 is coupled to a touchscreen 3065. The touch-screen controller 3055 detects contact andmovement on the touch screen 3065 using any of multiple touchsensitivity technologies. The other input controllers 3060 are coupledto other input/control devices, such as one or more buttons. Someembodiments include a near-touch sensitive screen and a correspondingcontroller that can detect near-touch interactions instead of or inaddition to touch interactions.

The memory interface 3010 is coupled to memory 3070. In someembodiments, the memory 3070 includes volatile memory (e.g., high-speedrandom access memory), non-volatile memory (e.g., flash memory), acombination of volatile and non-volatile memory, and/or any other typeof memory. As illustrated in FIG. 30, the memory 3070 stores anoperating system (OS) 3072. The OS 3072 includes instructions forhandling basic system services and for performing hardware dependenttasks.

The memory 3070 also includes communication instructions 3074 tofacilitate communicating with one or more additional devices; graphicaluser interface instructions 3076 to facilitate graphic user interfaceprocessing; image processing instructions 3078 to facilitateimage-related processing and functions; input processing instructions3080 to facilitate input-related (e.g., touch input) processes andfunctions; audio processing instructions 3082 to facilitateaudio-related processes and functions; and camera instructions 3084 tofacilitate camera-related processes and functions. The instructionsdescribed above are merely exemplary and the memory 3070 includesadditional and/or other instructions in some embodiments. For instance,the memory for a smartphone may include phone instructions to facilitatephone-related processes and functions. The above-identified instructionsneed not be implemented as separate software programs or modules.Various functions of the mobile computing device can be implemented inhardware and/or in software, including in one or more signal processingand/or application specific integrated circuits.

While the components illustrated in FIG. 30 are shown as separatecomponents, one of ordinary skill in the art will recognize that two ormore components may be integrated into one or more integrated circuits.In addition, two or more components may be coupled together by one ormore communication buses or signal lines. Also, while many of thefunctions have been described as being performed by one component, oneof ordinary skill in the art will realize that the functions describedwith respect to FIG. 30 may be split into two or more integratedcircuits.

B. Computer System

FIG. 31 conceptually illustrates another example of an electronic system3100 with which some embodiments of the invention are implemented. Theelectronic system 3100 may be a computer (e.g., a desktop computer,personal computer, tablet computer, etc.), phone, PDA, or any other sortof electronic or computing device. Such an electronic system includesvarious types of computer readable media and interfaces for variousother types of computer readable media. Electronic system 3100 includesa bus 3105, processing unit(s) 3110, a graphics processing unit (GPU)3115, a system memory 3120, a network 3125, a read-only memory 3130, apermanent storage device 3135, input devices 3140, and output devices3145.

The bus 3105 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 3100. For instance, the bus 3105 communicativelyconnects the processing unit(s) 3110 with the read-only memory 3130, theGPU 3115, the system memory 3120, and the permanent storage device 3135.

From these various memory units, the processing unit(s) 3110 retrievesinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments. Someinstructions are passed to and executed by the GPU 3115. The GPU 3115can offload various computations or complement the image processingprovided by the processing unit(s) 3110. In some embodiments, suchfunctionality can be provided using CoreImage's kernel shading language.

The read-only-memory (ROM) 3130 stores static data and instructions thatare needed by the processing unit(s) 3110 and other modules of theelectronic system. The permanent storage device 3135, on the other hand,is a read-and-write memory device. This device is a non-volatile memoryunit that stores instructions and data even when the electronic system3100 is off. Some embodiments of the invention use a mass-storage device(such as a magnetic or optical disk and its corresponding disk drive) asthe permanent storage device 3135.

Other embodiments use a removable storage device (such as a floppy disk,flash memory device, etc., and its corresponding drive) as the permanentstorage device. Like the permanent storage device 3135, the systemmemory 3120 is a read-and-write memory device. However, unlike storagedevice 3135, the system memory 3120 is a volatile read-and-write memory,such a random access memory. The system memory 3120 stores some of theinstructions and data that the processor needs at runtime. In someembodiments, the invention's processes are stored in the system memory3120, the permanent storage device 3135, and/or the read-only memory3130. For example, the various memory units include instructions forprocessing multimedia clips in accordance with some embodiments. Fromthese various memory units, the processing unit(s) 3110 retrievesinstructions to execute and data to process in order to execute theprocesses of some embodiments.

The bus 3105 also connects to the input and output devices 3140 and3145. The input devices 3140 enable the user to communicate informationand select commands to the electronic system. The input devices 3140include alphanumeric keyboards and pointing devices (also called “cursorcontrol devices”), cameras (e.g., webcams), microphones or similardevices for receiving voice commands, etc. The output devices 3145display images generated by the electronic system or otherwise outputdata. The output devices 3145 include printers and display devices, suchas cathode ray tubes (CRT) or liquid crystal displays (LCD), as well asspeakers or similar audio output devices. Some embodiments includedevices such as a touchscreen that function as both input and outputdevices.

Finally, as shown in FIG. 31, bus 3105 also couples electronic system3100 to a network 3125 through a network adapter (not shown). In thismanner, the computer can be a part of a network of computers (such as alocal area network (“LAN”), a wide area network (“WAN”), or an Intranet,or a network of networks, such as the Internet. Any or all components ofelectronic system 3100 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself. In addition, someembodiments execute software stored in programmable logic devices(PLDs), ROM, or RAM devices.

As used in this specification and any claims of this application, theterms “computer”, “server”, “processor”, and “memory” all refer toelectronic or other technological devices. These terms exclude people orgroups of people. For the purposes of the specification, the termsdisplay or displaying means displaying on an electronic device. As usedin this specification and any claims of this application, the terms“computer readable medium,” “computer readable media,” and “machinereadable medium” are entirely restricted to tangible, physical objectsthat store information in a form that is readable by a computer. Theseterms exclude any wireless signals, wired download signals, and anyother ephemeral signals.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. For instance, many of the figuresillustrate various touch gestures (e.g., taps, double taps, swipegestures, press and hold gestures, etc.). However, many of theillustrated operations could be performed via different touch gestures(e.g., a swipe instead of a tap, etc.) or by non-touch input (e.g.,using a cursor controller, a keyboard, a touchpad/trackpad, a near-touchsensitive screen, etc.). In addition, a number of the figures (includingFIGS. 2, 5, 6, 7, 10, 12, 14, 18, 23, and 27) conceptually illustrateprocesses. The specific operations of these processes may not beperformed in the exact order shown and described. The specificoperations may not be performed in one continuous series of operations,and different specific operations may be performed in differentembodiments. Furthermore, the process could be implemented using severalsub-processes, or as part of a larger macro process. Thus, one ofordinary skill in the art would understand that the invention is not tobe limited by the foregoing illustrative details, but rather is to bedefined by the appended claims.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. For example, controls for setting thevarious adjustment settings as slider and numerical controls in FIG. 11.The sliders of such embodiments provide a visual indication of a settingvalue as a knob is slid along the slider to set a value for the slider.However, in some embodiments, the slider controls shown in any of thosefigures could be replaced with any other control capable of receiving avalue (e.g., a single value), such as a vertical slider control, a pulldown menu, a value entry box, an incremental tool activated by keyboardkeys, other range related UI controls (e.g., dials, buttons, numberfields, and the like), etc. Similarly, the slider controls shown in thefigures are either depicted as being set with a finger gesture (e.g.,placing, pointing, tapping one or more fingers) on a touch sensitivescreen or simply shown in a position without any indication of how theywere moved into position. One of ordinary skill in the art willunderstand that the controls of FIG. 11 and the tonal curve points ofFIG. 17 can also be activated and/or set by a cursor control device(e.g., a mouse or trackball), a stylus, keyboard, a finger gesture(e.g., placing, pointing, tapping one or more fingers) near a near-touchsensitive screen, or any other control system in some embodiments. Thus,one of ordinary skill in the art would understand that the invention isnot to be limited by the foregoing illustrative details, but rather isto be defined by the appended claims.

What is claimed is:
 1. A method of automatically adjusting shadow levelsof an image, the method comprising: generating a set of histograms thatidentify different characteristics of the image, wherein the set ofhistograms comprises a structure histogram comprising values based onrelationships between each particular pixel of a plurality of pixels inthe image and the neighboring pixels of the particular pixel;determining a shadow lift setting level based on a set of statisticscalculated for the set of histograms; and applying a shadow lift at thedetermined shadow lift setting level to the image, wherein thegenerating, determining and applying are performed by an electronicdevice.
 2. The method of claim 1, wherein the relationships on which thestructure histogram values are based comprise differences betweenluminance values of the particular pixels and the neighboring pixels ofthe particular pixels.
 3. The method of claim 1, wherein a relationshipbetween a particular pixel and a neighboring pixel contributes to avalue of the structure histogram only when a luminance value of theparticular pixel is different from a luminance value of the neighboringpixel.
 4. The method of claim 1, wherein the set of histograms furthercomprises a luminance histogram.
 5. The method of claim 1, wherein theset of histograms further comprises a cumulative luminance histogram. 6.The method of claim 1 further comprising: determining an ISO value ofthe image; and reducing the applied shadow lift setting level based onthe ISO value.
 7. The method of claim 6, wherein reducing the appliedshadow lift setting level based on the ISO value comprises lowering theshadow lift setting value more for higher ISO values than for lower ISOvalues.
 8. An electronic device comprising: a set of processing units;and a non-transitory machine-readable medium storing a program whichwhen executed by at least one of the processing units enhances an imagecomprising a plurality of pixels, the program comprising sets ofinstructions for: identifying a first subset of the plurality of pixelsthat comprises skin-tone colored pixels; and applying a variable gammaadjustment to only a second subset of the plurality of pixels thatexcludes the first subset of pixels, the variable gamma adjustmentcomprising an adjustment that raises at least one value of each pixel toa power that varies based on characteristics of the pixel.
 9. Theelectronic device of claim 8, wherein the program further comprises aset of instructions for applying the variable gamma adjustment to thefirst subset of pixels unless a face is identified in the image.
 10. Theelectronic device of claim 8, wherein the variable gamma adjustment ispartly based on a user setting level, the program further comprising aset of instructions for applying the variable gamma adjustment to thefirst subset of pixels when the user setting level is above a presetthreshold setting level.
 11. The method electronic device of claim 8,wherein when a particular characteristic of the image is detected, aconstant gamma adjustment is applied to the first subset of pixels. 12.A method of generating a formula for setting shadow lift levels of newimages, the method comprising: receiving data for a plurality of imagesthat indicates a user-specified maximum acceptable shadow lift settingfor each of the plurality of images; for each of the images calculatinga set of statistics based on a set of histograms generated for theimage; and with an electronic device, performing a mathematicalregression to generate the formula for setting shadow lift levels of newimages based on the calculated sets of statistics and the user-specifiedmaximum acceptable shadow lift settings for the plurality of images. 13.The method of claim 12, wherein the set of histograms comprises astructure histogram, a luminance histogram, and a cumulative luminancehistogram.
 14. The method of claim 13, wherein the structure histogramis generated based on both luminance values and an arrangement of pixelsin the image.
 15. A non-transitory machine readable medium storing aprogram that when executed by at least one processing unit automaticallyadjusts shadow levels of an image, the program comprising sets ofinstructions for: generating a set of histograms that identify differentcharacteristics of the image, wherein the set of histograms comprises astructure histogram comprising values based on relationships betweeneach particular pixel of a plurality of pixels in the image and theneighboring pixels of the particular pixel; determining a shadow liftsetting level based on a set of statistics calculated for the set ofhistograms; and applying a shadow lift at the determined shadow liftsetting level to the image.
 16. The non-transitory machine readablemedium of claim 15, wherein the relationships on which the structurehistogram values are based comprise differences between luminance valuesof the particular pixels and the neighboring pixels of the particularpixels.
 17. The non-transitory machine readable medium of claim 15,wherein the set of histograms further comprises a histogram thatmeasures the luminance values of the pixels in the image.
 18. Thenon-transitory machine readable medium of claim 17, wherein the set ofhistograms further comprises a cumulative luminance histogram.
 19. Thenon-transitory machine readable medium of claim 15, wherein the set ofhistograms further comprises a cumulative luminance histogram thatmeasures how many pixels have a given luminance or lower.
 20. Thenon-transitory machine readable medium of claim 15, wherein the programfurther comprises sets of instructions for: reading an ISO value frommetadata of the image to determine an ISO value of the image; andreducing the applied shadow lift setting level based on the ISO value.21. A non-transitory machine readable medium storing a program which,when executed by at least one processing unit, enhances an imagecomprising a plurality of pixels, the program comprising sets ofinstructions for: identifying a first subset of the plurality of pixelsthat comprises skin-tone colored pixels; and applying a variable gammaadjustment to only a second subset of the plurality of pixels thatexcludes the first subset of pixels, the variable gamma adjustmentcomprising an adjustment that raises at least one value of each pixel toa power that varies based on characteristics of the pixel.
 22. Thenon-transitory machine readable medium of claim 21, wherein the programfurther comprises a set of instructions for applying the variable gammaadjustment to the first subset of pixels unless a face is identified inthe image.
 23. The non-transitory machine readable medium of claim 22,wherein the variable gamma adjustment is partly based on a user setting,wherein the program further comprises a set of instructions for applyingthe variable gamma adjustment to the first subset of pixels when theuser setting level is above a preset threshold setting level.
 24. Thenon-transitory machine readable medium of claim 21, wherein the programfurther comprises a set of instructions for applying a constant gammaadjustment to the first subset of pixels when a particularcharacteristic of the image is detected.
 25. The non-transitory machinereadable medium of claim 21, wherein the variable gamma adjustment for aparticular pixel is based partly on a luminance value of a correspondingpixel in a blurred version of the image.